roman
2018-01-02 10:54
has joined #devops
oskarth
2018-01-02 10:54
has joined #devops
oskarth
2018-01-02 10:54
you were early
ivan.borovkov
2018-01-02 10:54
has joined #devops
roman
2018-01-02 10:55
> So:
> 1. Solve status-react issue #2566 - acceptance criteria for v1: show build success or fail in Github PR, that's it
> 2. For PR builds, make step in pipeline before Jenkinsfile that check if Jenkinsfile has been modified and prompts/denies build by default. Also make issue in status-react for this.
roman
2018-01-02 10:56
> ok, so what exactly we want here https://github.com/status-im/status-react/issues/2566 ?
> 1. build branch on demand (probably by comment in PR form anyone with write permissions in repo)
> 2. if build is successful add comment to PR with Android/iOS diawi links. Also maybe we want to share ios archive with instructions how to sign it locally, in case if someone wants to run app on real ios device. If that's possible at all.
> 3. If build fails lets share the link to jenkins log (make sure that it's public, not necessarily located on jenkins at all) and probably add last N lines of log to PR as comment.
oskarth
2018-01-02 10:56
^ @ivan.borovkov
roman
2018-01-02 10:57
@ivan.borovkov so that's how we see priorities atm:
1. https://github.com/status-im/status-react/issues/2566
2. releases deployment
3. google auth on jenkins
4. SOB ansible/docker
roman
2018-01-02 10:58
above is our discussion about 1. https://github.com/status-im/status-react/issues/2566
roman
2018-01-02 10:58
so, my initial thoughts are here https://status-im.slack.com/archives/C8M1Z25J8/p1514890568000161
roman
2018-01-02 10:58
that's what @oskarth proposes as initial part of task
https://status-im.slack.com/archives/C8M1Z25J8/p1514890531000129
cryptowanderer
2018-01-02 10:58
has joined #devops
jeluard
2018-01-02 11:00
has joined #devops
carl
2018-01-02 11:01
has joined #devops
roman
2018-01-02 11:06
> > 1. Solve status-react issue #2566 - acceptance criteria for v1: show build success or fail in Github PR, that's it
> i think we have this now, though as we don't build PRs from external contributors it always fails
> maybe let?s start from building branch by some comment?
so that's the last copy of messages here, i hope you will be able to sort out in this mess @ivan.borovkov, just ask question if not :slightly_smiling_face:
naghdy
2018-01-02 11:10
has joined #devops
ivan.borovkov
2018-01-02 11:23
@roman thanks for clarifying this
ivan.borovkov
2018-01-02 11:24
Regarding monitoring jenkinsfile for changes, are you sure we want to monitor only this file? We dont execute any other scripts from there? Or we dont execute anything that can have injected code that will harm us?
roman
2018-01-02 11:25
we also execute this one at least https://github.com/status-im/status-react/blob/develop/ios/scripts/set_xcode_version.sh
roman
2018-01-02 11:25
about injections, that's a good question, there could be vulnerabilities
roman
2018-01-02 11:26
like even if person changes the version of some npm dependency i can imagine scenario when during `npm install` it also executes something harmful
ivan.borovkov
2018-01-02 11:27
exactly
roman
2018-01-02 11:27
idk probably we could execute npm install in container or so
roman
2018-01-02 11:27
and then don't care about consequences
ivan.borovkov
2018-01-02 11:27
we can isolate this environment, for example separate build machine for external contributions
ivan.borovkov
2018-01-02 11:28
or we can try building in container, or new vms
ivan.borovkov
2018-01-02 11:28
i?m not sure how it could be implemented for IOS, need some r&d
roman
2018-01-02 11:28
whatever you see as the best solution for us
roman
2018-01-02 11:29
also services like travis/circleci were mentioned, but i suspect they will be slow
ivan.borovkov
2018-01-02 11:29
they are fast enough on paid tarifs
ivan.borovkov
2018-01-02 11:29
and none of them offer osx on free plan
ivan.borovkov
2018-01-02 11:31
however there are still ways to escape docker environment and go to main host..
oskarth
2018-01-02 11:58
maybe we can have a whitelist for core contributors and only allow npm/jenkins upgrades from such users? dunno
oskarth
2018-01-02 11:58
maybe not practical since bounties also do this, but then we have presumably reviewed stuff
oskarth
2018-01-02 11:59
should also consider threat model, is it installing a bitcoin miner? or taking our certificates? if latter, maybe we can just isolate TF / release builds from PR builds
ivan.borovkov
2018-01-02 12:14
By modifying scripts basically someone will have full access to the system, he can add his ssh keys, send somewhere certs, erase system, install miners, add app that will act as a part of botnet, and lots of other things
ivan.borovkov
2018-01-02 12:15
We need to understand first how to protect ourselves from this, and then proceed with builds from everyone
oskarth
2018-01-02 12:19
isn't this assuming automated builds with no review and manual trigger before? as long as we have this we don't need to worry that much, right? at least as a first step
oskarth
2018-01-02 12:21
even if they do all these things it's really bad obviously but not anywhere near as terrible as it would be at a normal company, with user data, critical systems, payments or w/e.
igor680
2018-01-02 14:43
has joined #devops
pablo821
2018-01-02 14:58
has joined #devops
dmitryn
2018-01-02 15:24
has joined #devops
themue
2018-01-02 18:21
has joined #devops
ivan.borovkov
2018-01-02 18:33
I think it is good way to do this, but it will require some java development. It could be easily outsourced, and then kept as external plugin, or pull-requested to initial repository, as i think this is good point for OSS.
https://github.com/jenkinsci/github-branch-source-plugin
This plugin is implementing most of this logic, we just need to add logic that will check approval from team member before build.
How is outsourcing implemented in Status?
I?m not java developer, but i think it will be great to have this implemented in that plugin.
meanwhile i will do some r&d about #2 and #3
noman
2018-01-02 18:56
has joined #devops
oskarth
2018-01-03 08:44
As far as I can tell, these are identical:
https://jenkins.status.im/job/status-react/job/pull%20requests/job/bugfix%252Fquickfix-disable-cljs-test-jenkins/1/console
https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-2873/
except one is working and one isn't, so diff Jenkins config.
Do you know what's up with this @ivan.borovkov? Is it a WIP:
ivan.borovkov
2018-01-03 08:46
Basically it should not try to build branch and build only PR according to settings
oskarth
2018-01-03 08:49
so can we kill the old configuration? a bit confusing right now
oskarth
2018-01-03 08:56
@ivan.borovkov Parameterized builds for rebasing:
Goal: Allow QA to test PRs that require trivial rebasing without manual intervention from contributor. Do this with parameterized builds in Jenkins.
1. Jenkinsfile parameterized config, manually add T/F in UI
2. Jenkinsfile.parameters (in git to keep it up to date, but it's also inline at Jenkins so need to change x2):
Something like (in bash/groovy shell) `IF REBASE_DEVELOP => git rebase origin` - test manually in CLI with some trivial change first and make sure it is non interactive. This can possibly be another (conditional) pipeline step, or just another shell command.
ivan.borovkov
2018-01-03 08:57
Maybe it is worth to use jenkinsfile from git for parametrized build? Why it is being copied?
oskarth
2018-01-03 08:58
Agreed, because I couldn't figure out how to do it with parameterized build/multipipeline stuff, I'm sure it is possible
oskarth
2018-01-03 08:58
I just timeboxed it and that's the best I could do in 1-2h
oskarth
2018-01-03 08:59
also some stuff re credentials and exact source of it (since we parameterize it on on branch and remote, where do we take jenkinsfile from?) etc - if you want to experiment go ahead, but maybe a good idea to clone the pipeline and do it separately so it doesn't impact QA
ivan.borovkov
2018-01-03 09:06
I've changed it to use Jenkinsfile.parameters from github
oskarth
2018-01-03 09:07
you sure it works correctly?
ivan.borovkov
2018-01-03 09:07
but i think github is missing few options: https://jenkins.status.im/job/status-react/job/parametrized%20build/215/console
oskarth
2018-01-03 09:07
I had issues with it
ivan.borovkov
2018-01-03 09:14
`test $REBASE_DEVELOP -eq 1 && git rebase origin`
can you add on the step when it needed?
ivan.borovkov
2018-01-03 09:14
parametrized build is using jenkinsfile now from github so no need to edit it twice
oskarth
2018-01-03 09:17
something like this yes
oskarth
2018-01-03 09:17
you mean you want me to change Jenkinsfile in PR?
oskarth
2018-01-03 09:18
I think it isn't $REBASE_DEVELOP btw, $ is bash, this is groovy variables https://github.com/status-im/status-react/blob/develop/Jenkinsfile.parameters#L26
ivan.borovkov
2018-01-03 09:21
it should be fine like this:
```sh 'test ${REBASE_DEVELOP} -eq 1 && git rebase origin'```
But need to test
oskarth
2018-01-03 09:22
ok, is there something you want me to do or can you test?
ivan.borovkov
2018-01-03 09:23
Yes, please test, i?m still not sure how to check it is working correctly
ivan.borovkov
2018-01-03 09:24
basically jenkinsfile allows developers to define how application need to be built, you definitely know this better than me
oskarth
2018-01-03 09:25
yeah ok sure, will do soonish
oskarth
2018-01-03 09:26
so to test I should do this in git now but not in Jenkins, correct?
oskarth
2018-01-03 09:27
and it will use Jenkinsfile.parameter, from where exactly? from develop? how does it work when you parameterize a build from remote/branch?
ivan.borovkov
2018-01-03 09:28
It should use Jenkinsfile.parameters from the branch you are requesting to build, but if needed we can specify branch
oskarth
2018-01-03 09:29
ok cool, will try
oskarth
2018-01-03 10:42
@ivan.borovkov seems like all parameterized builds failed last day or so, https://jenkins.status.im/job/status-react/job/parametrized%20build/ also seems due to some Jenkins cfg stuff - any idea why?
ivan.borovkov
2018-01-03 10:44
I've changed it only today, earlier was doing changes just for Jenkinsfile
oskarth
2018-01-03 10:50
fixing 219
oskarth
2018-01-03 10:53
problem seems to be LOG_LEVEL_STATUS_GO wasn't defined in UI. I added it now. Do you know if it is possible to default with groovy syntax in case variable isn't defined? in bash this would just be "" by default I think
oskarth
2018-01-03 11:30
New issue re Jenkins reliability, more medium-term: https://github.com/status-im/status-react/issues/2878
ivan.borovkov
2018-01-03 11:37
Anyone changed parametrized builds back to pipeline script instead of pipeline script from scm?
oskarth
2018-01-03 11:38
no, I added a new env variable in UI which meant clicking save/apply
oskarth
2018-01-03 11:38
but I think I did this after your changes, and I did'nt touch the jenkinsfile location at all
ivan.borovkov
2018-01-03 11:38
but it is back to pipeline script
oskarth
2018-01-03 11:38
yeah I have no idea
oskarth
2018-01-03 11:38
I didn't touch pipelinescript/scm source
oskarth
2018-01-03 16:01
More CI stuff @ivan.borovkov, not super high prio but FYI: We currently have Travis as well as Jenkins for status-go. Due to some linting stuff. It'd probably be better of we only use Jenkins, so it'd be nice to move status-go Travis checks to Jenkins at some point. (We are more invested and dependent on Jenkins, largely due to OSX builds etc).
adamb
2018-01-03 16:02
has joined #devops
oskarth
2018-01-03 16:02
@adamb has more context, naive search shows this recentish commit e.g. https://github.com/status-im/status-go/commit/d6d5945ab489e78f385df706655475a6fc2a93e7
adamb
2018-01-03 16:05
status-go-manual should be a pipeline from scm. Let me prepare a PR in status-go first as there are differences
adamb
2018-01-03 16:09
Actually, there are no differences.
ivan.borovkov
2018-01-03 16:12
Sorry for confusion that was about status-react repo.
oskarth
2018-01-05 07:00
@ivan.borovkov @roman Jenkins nightly failing again.... https://jenkins.status.im/job/status-react/job/nightly/158/console
oskarth
2018-01-05 07:00
```
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/status-im/status-react.git" returned status code 255:
stdout:
stderr: error: could not lock config file .git/config: No space left on device
```
oskarth
2018-01-05 07:04
Created a devops board since there are a lot of loose threads, and I know this unstructuredness was something that was blocking you earlier @ivan.borovkov https://github.com/orgs/status-im/projects/7
I'd just like to re-iterate the importance of https://github.com/status-im/status-react/issues/2878
oskarth
2018-01-05 07:05
gonna try to find jenkins machine as we literally can't do release candidates or testing while this is happening, so it's a flaming fire kind of thing
ivan.borovkov
2018-01-05 07:10
doing cleanup on macos1 and also changed settings to keep old builds on jenkins for 2 days
oskarth
2018-01-05 07:10
great, thank you
ivan.borovkov
2018-01-05 07:10
83G available, should be able to build now
oskarth
2018-01-05 07:11
not ending straight away so seems good https://jenkins.status.im/job/status-react/job/nightly/, thanks for speedy fix
ivan.borovkov
2018-01-05 07:15
there is status-cluster project with my tasks, so maybe it is worth to create this board there
oskarth
2018-01-05 07:17
project is org-wide, so you can add issues from status-cluster
ivan.borovkov
2018-01-05 07:17
i see, thanks
yenda
2018-01-05 10:57
has joined #devops
ivan.borovkov
2018-01-08 08:06
@oskarth @roman hi. what do you think about this? (#1)
oskarth
2018-01-08 08:08
@ivan.borovkov
> 1. build branch on demand (probably by comment in PR form anyone with write permissions in repo)
sure, sounds good to me.
btw, try not to highlight Roman unless it is emergency, better if we create issues / mention in GH, trying to get his time spent on specific things in code that only he knows about as much as possible :slightly_smiling_face:
ivan.borovkov
2018-01-08 08:13
Ok, will not highlight him. Do you know how i can initiate #1 dev outsourcing? Or maybe someone from the team can help with this?
oskarth
2018-01-08 08:21
we can create a bounty :slightly_smiling_face: do you want to try create issue describing problem and solution you imagine in detail? can ping me and others then to look over before putting actual bounty on it. https://www.youtube.com/watch?v=Qbv7rs6gwcI I made a tutorial-ish
oskarth
2018-01-08 08:24
Reason for status-react: it is open repo, even if it doesn't 100% belong there. I guess one problem is also how open this task is, ideally you can help person who fixes it locally getting it tested/integrated etc
roman
2018-01-08 09:07
`can't write to output file (No space left on device)` on jenkins
roman
2018-01-08 09:08
@ivan.borovkov
ivan.borovkov
2018-01-08 09:08
checking
ivan.borovkov
2018-01-08 09:22
should be enough space, i will reconfigure jobs to delete artifacts earlier
oskarth
2018-01-08 09:24
maybe we want to consider putting artifacts on S3 or something like this
ivan.borovkov
2018-01-08 09:25
we have them uploaded to dawi. Do we need to double store them?
oskarth
2018-01-08 09:26
I guess I mean to get rid of disk space issue once and for all
oskarth
2018-01-08 09:26
but maybe this isn't reasonable, not sure how rolling buffer etc works
ivan.borovkov
2018-01-08 09:43
I've reconfigured artifacts keep policies, should be better now
roman
2018-01-08 12:30
`fatal: bad revision '^'` again
ivan.borovkov
2018-01-08 12:33
But it is checking out develop normally
roman
2018-01-08 12:39
let?s try again
roman
2018-01-08 13:07
and again
ivan.borovkov
2018-01-08 13:42
Failing this part of code:
```# Bundle version (commits-on-master[-until-branch "." commits-on-branch])
# Assumes that two release branches will not diverge from the same commit on master.
if [ $(git rev-parse --abbrev-ref HEAD) = "$MAIN_GIT_BRANCH" ]; then
MASTER_COMMIT_COUNT=$(git rev-list --count HEAD)
BRANCH_COMMIT_COUNT=0
BUNDLE_VERSION="$MASTER_COMMIT_COUNT"
else
MASTER_COMMIT_COUNT=$(git rev-list --count $(git rev-list ${MAIN_GIT_BRANCH}.. | tail -n 1)^)
BRANCH_COMMIT_COUNT=$(git rev-list --count ${MAIN_GIT_BRANCH}..)
if [ $BRANCH_COMMIT_COUNT = 0 ]
then BUNDLE_VERSION="$MASTER_COMMIT_COUNT"
else BUNDLE_VERSION="${MASTER_COMMIT_COUNT}.${BRANCH_COMMIT_COUNT}"
fi
fi```
Since you are trying to build develop branch, variable $MAIN_GIT_BRANCH contains develop.
https://github.com/status-im/status-react/blob/develop/ios/scripts/set_xcode_version.sh#L60
on this line `git rev-list ${MAIN_GIT_BRANCH}.. | tail -n 1` returns 0 output, which is not expected to be as
```bash-3.2$ git rev-list --count ^
fatal: bad revision '^'```
ivan.borovkov
2018-01-08 13:43
This logic need to be reworked, i will take a look on it
ivan.borovkov
2018-01-08 13:46
I've created issue for this https://github.com/status-im/status-cluster/issues/26 feel free to drop comments there if you have any
ivan.borovkov
2018-01-08 13:48
I think it should not be `${MAIN_GIT_BRANCH}..` and should be `${MAIN_GIT_BRANCH}` on lines 60 and 61
ivan.borovkov
2018-01-08 13:48
will test it
ivan.borovkov
2018-01-08 13:50
yes, this lines should be something like this
```
MASTER_COMMIT_COUNT=$(git rev-list --count $(git rev-list ${MAIN_GIT_BRANCH} | tail -n 1))
BRANCH_COMMIT_COUNT=$(git rev-list --count ${MAIN_GIT_BRANCH})
```
ivan.borovkov
2018-01-08 13:51
it works for me in console
ivan.borovkov
2018-01-08 14:30
are you sure you want to build develop branch from PR pipeline? There are parametrized and nightly pipelines that can build it without any problems
roman
2018-01-08 15:26
`PR-2940 build started.` @ivan.borovkov we need to fing the way to show real branch name in #internal-jenkins, it's very hard to keep all these numbers in mind
roman
2018-01-08 15:28
something like this `git rev-parse --abbrev-ref HEAD`
roman
2018-01-08 15:47
nope this will not work
ivan.borovkov
2018-01-08 15:47
yes, it wont
ivan.borovkov
2018-01-08 15:48
One of the options that i?ve found so far is to get POST form github into environment variables, and then use pull_request.head.ref to send notification to slack
ivan.borovkov
2018-01-08 15:49
Also i?ve found that github plugin should push this info to env variables, but i havent found any documentation about this
roman
2018-01-08 15:52
The plugin makes some very useful environment variables available.
ghprbActualCommit
ghprbActualCommitAuthor
ghprbActualCommitAuthorEmail
ghprbPullDescription
ghprbPullId
ghprbPullLink
ghprbPullTitle
ghprbSourceBranch
ghprbTargetBranch
ghprbCommentBody
sha1
roman
2018-01-08 15:52
i would print there two and check
ghprbSourceBranch
ghprbTargetBranch
roman
2018-01-08 15:52
or all vars
roman
2018-01-08 15:53
ivan.borovkov
2018-01-08 15:53
we are using this https://wiki.jenkins.io/display/JENKINS/GitHub+Branch+Source+Plugin
roman
2018-01-08 15:55
can we print them all? :slightly_smiling_face:
anna
2018-01-08 15:57
has joined #devops
anna
2018-01-08 15:59
hey :slightly_smiling_face: why all builds in https://jenkins.status.im/job/status-react/job/pull%20requests/ are red?
ivan.borovkov
2018-01-08 16:03
ivan.borovkov
2018-01-08 16:03
seems to be that CHANGE_BRANCH is what we need
ivan.borovkov
2018-01-08 16:04
Please refer to ?Pull requests? tab
roman
2018-01-08 16:06
yes, nice :slightly_smiling_face:
ivan.borovkov
2018-01-08 16:08
I will issue PR to jenkinsfile to display it
ivan.borovkov
2018-01-08 16:20
looks like it is working
oskarth
2018-01-08 17:17
can we get rid of old tab?
oskarth
2018-01-09 07:35
Hey @ivan.borovkov! which issue are you currently working on? Don't see anything in progress at https://github.com/orgs/status-im/projects/7
ivan.borovkov
2018-01-09 07:40
I've moved to progress main task that i?m working on
ivan.borovkov
2018-01-09 07:40
and will comment others
oskarth
2018-01-09 07:52
thanks
oskarth
2018-01-09 10:11
Good stuff at build speed @ivan.borovkov. Do you also think there is something we can do to simplify the build steps? I think this would also help with https://github.com/status-im/status-react/issues/2878 As in fewer of them / more robust (yarn is one example)
yenda
2018-01-09 11:16
@oskarth you mention build speed improvement could it be related to the fact that the android build is not functionnal anymore ?
ivan.borovkov
2018-01-09 11:19
yes, i replied in that thread
yenda
2018-01-09 11:19
exemple here: https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-2953/ build #2 succeeded but apk is not working on android device
ivan.borovkov
2018-01-09 11:35
Dont know about this, need to be tested, measured. Top ways seems to be not working. By the way why do we need to increase speed of nightly build? It runs at night, when noone will be able to see it
oskarth
2018-01-09 11:43
speed: my thinking is that nightly is the benchmark, I think this will impact PR etc builds too
oskarth
2018-01-09 11:43
the general goal is to move towards continuous delivery, so we get feedback quickly after PR, have automated tests etc and can merge with more confidence
oskarth
2018-01-09 11:43
and only testing more specific flows / general release stuff
oskarth
2018-01-09 11:43
but this is more long term horizon
oskarth
2018-01-09 11:47
also quick build time leads to qualitatively different work, like devs can use CI build in day to day testing, local builds might be faster if we get build steps more robust (vendor/yarn/more) etc
oskarth
2018-01-09 12:20
dont know
oskarth
2018-01-09 12:20
I haven't made any changes related to speed stuff
roman
2018-01-09 14:55
@ivan.borovkov it would be great to find a solution for situation when we push commits frequently and a new build added to the queue when the previous one is not finished. It would be great just to skip/abort older builds
ivan.borovkov
2018-01-09 15:03
I will try migrating to https://github.com/jenkinsci/ghprb-plugin instead of current github plugin, it seems to be solves this and some other problems. Will do config on separate folder to make sure that it doesnt break a lot.
roman
2018-01-09 15:09
nice, thank you
roman
2018-01-09 15:21
https://status-im.slack.com/archives/C8KB9DR60/p1515510716000600
```
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)
```
ivan.borovkov
2018-01-09 15:31
restarted build, it seems to be working fine. I've noticed that this happened when there were 4 concurrent builds on that server. Changed max number of concurrent builds to 3, should not be killed anymore in case of big load
tatiana
2018-01-09 16:39
has joined #devops
oskarth
2018-01-11 04:59
@ivan.borovkov flaming fire again :slightly_smiling_face:
fatal error: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/libtool: can't write to output file (No space left on device)
https://jenkins.status.im/job/status-react/job/nightly/173/console nightly
oskarth
2018-01-11 06:32
I'm on box via VNC, slow af, did you get SSH setup for it?
oskarth
2018-01-11 06:32
Also last time I did clean up I did some nasty things, dunno if you have found more proper way to clean it
oskarth
2018-01-11 06:38
Not sure about admin pw either
oskarth
2018-01-11 06:39
now it says 40GB avail, just recently it said 600mb - is this a cleanup job or smt?
ivan.borovkov
2018-01-11 06:59
i?m doing some cleanup
ivan.borovkov
2018-01-11 07:00
now it should have some space
ivan.borovkov
2018-01-11 07:00
will think how to cleanup on regular basis
oskarth
2018-01-11 07:01
ok
oskarth
2018-01-11 07:02
I think we should treat this box as temporary memory/cache kind
oskarth
2018-01-11 07:02
and if there are builds or whatever we want to keep they can be stored on S3 or w/e, maybe gzipped up or something
pedro
2018-01-13 11:54
has joined #devops
oskarth
2018-01-13 11:55
Jenkins develop issues https://jenkins.status.im/job/status-react/job/nightly/
```
[nightly] Running shell script
+ mv /Users/jenkins/workspace/status-react/nightly/android/app/build/outputs/apk/app-release.apk /Users/jenkins/workspace/status-react/nightly/android/app/build/outputs/apk/im.status.ethereum-c8ef43.apk
mv: rename /Users/jenkins/workspace/status-react/nightly/android/app/build/outputs/apk/app-release.apk to /Users/jenkins/workspace/status-react/nightly/android/app/build/outputs/apk/im.status.ethereum-c8ef43.apk: No such file or directory
[Pipeline] }
[Pipeline] // stage
[Pipeline] slackSend
run slackstepsend, step null:false, desc BZU9o3LpIcV1ST0RFDhXjh7L:true
Slack Send Pipeline step configured values from global config - baseUrl: true, teamDomain: true, token: true, channel: true, color: false
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE
```
oskarth
2018-01-13 11:56
Unless it is a Jenkins issue (@ivan.borovkovthought it was OOM error), suspects imo are:
- https://github.com/status-im/status-react/pull/2844
- https://github.com/status-im/status-react/pull/2831 (cc @pedro)
But CI passed for both those PRs, so dunno why it'd work there but not in develop.
oskarth
2018-01-13 11:58
> It is possible that new version is producing output to different directory, i?ve found app-release.apk in release/ subdirectory there. So most probably it is required to modify that script that renames build -- v2nek
oskarth
2018-01-13 11:59
do you want to edit the Jenkinsfile with PR @ivan.borovkov? if you can see where it should be
oskarth
2018-01-13 11:59
Just one line in ` stage('Deploy (Android)') {` I think
oskarth
2018-01-13 12:04
Eh, looks like 2844 succeeded by accident:
```
[react_pull_requests_PR-2844-5AUROJC26LUPXLHJXYU5BYGUT6EEAHFTHBRQA6RRMH7MVVF2KI5Q] Running shell script
+ curl https://upload.diawi.com/ -F token=**** -F file=@android/app/build/outputs/apk/app-release.apk -F find_by_udid=0 -F wall_of_apps=0
+ jq -r .job
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
curl: (26) couldn't open file "android/app/build/outputs/apk/app-release.apk"
[Pipeline] sh
[react_pull_requests_PR-2844-5AUROJC26LUPXLHJXYU5BYGUT6EEAHFTHBRQA6RRMH7MVVF2KI5Q] Running she
```
oskarth
2018-01-13 12:05
ok let me revert, probably this one
ivan.borovkov
2018-01-13 12:07
gradle4.1 changed output directory for final apk, which seems to be cannot be modified.
ivan.borovkov
2018-01-13 12:07
so it should be simple just to modify path to apk, add `release/`
oskarth
2018-01-13 12:07
please approve
oskarth
2018-01-13 12:07
ok if you can see
oskarth
2018-01-13 12:07
it and it is crystal clear on box what to change, please issue PR that fixes Jenkinsfile
oskarth
2018-01-13 12:08
otherwise defaulting to revert bad commit
oskarth
2018-01-13 12:09
which do you prefer?
oskarth
2018-01-13 12:10
if PR let me know and I'll approve, if revert please approve so we can merge
oskarth
2018-01-13 12:10
going afk soon unfortunately
ivan.borovkov
2018-01-13 12:11
ill push pr now
oskarth
2018-01-13 12:11
ok, cheers
oskarth
2018-01-13 12:13
approved
oskarth
2018-01-13 12:13
thanks :slightly_smiling_face:
oskarth
2018-01-13 12:13
merging
oskarth
2018-01-13 12:14
do you have any idea why Jenkins would say that PR was successful despite there being the same issue in that one?
ivan.borovkov
2018-01-13 12:14
we have problems with automated pr testing :slightly_smiling_face:
ivan.borovkov
2018-01-13 12:15
ill add space cleanup option before building
ivan.borovkov
2018-01-13 12:15
for all builds
oskarth
2018-01-13 12:15
ok, so maybe we need some sanity check step where we run actual build on device
oskarth
2018-01-13 12:15
sounds good
ivan.borovkov
2018-01-13 12:15
(if there is such possibility)
oskarth
2018-01-13 12:15
yeah not sure
ivan.borovkov
2018-01-13 12:18
added clean before checkout for pull requests
ivan.borovkov
2018-01-13 12:20
aalso added it to nightly to ensure that workspace is clean
oskarth
2018-01-13 12:21
Great
ivan.borovkov
2018-01-13 12:42
Should we deliver nightly android build to diawi same as all other builds?
oskarth
2018-01-13 12:45
Think so, dunno if @anna has different opinion
anna
2018-01-14 12:19
I don't think it's necessary - apk link in http://artifacts.status.im:8081/artifactory/nightlies-local/ should be ok, unless @adan has some preferences for autiomation
adan
2018-01-14 12:26
has joined #devops
adan
2018-01-14 12:28
I am okay with current approach of delivering nightly builds
adan
2018-01-14 12:29
however would be great to have an ability to get nightly builds via API call to artifactory
adan
2018-01-14 12:30
It's should be easy to setup, https://www.jfrog.com/confluence/display/RTF/Artifactory+REST+API
adan
2018-01-14 12:30
see 'All Builds' request, for now we can't make public requests to artifacotry
adan
2018-01-14 12:32
however since our nightly builds are public, it should be okay to have public 'get/builds' request
adan
2018-01-14 12:32
and it will help me much, since for now I am using regex in order to find specific build :disappointed:
adan
2018-01-14 13:33
@ivan.borovkov 3 nightly builds in a row were not published to http://artifacts.status.im:8081/artifactory/nightlies-local/
adan
2018-01-14 13:33
are you working on something right now?
adan
2018-01-14 13:38
404 file not found, after clicking on the nightly link (https://status-im.slack.com/archives/C8RNGLKRT/p151593655900004)
ivan.borovkov
2018-01-14 13:40
@adan https://github.com/status-im/status-react/pull/3032 this will fix it when merged to develop
roman
2018-01-15 11:54
`Jenkins is going to shut down` @ivan.borovkov is this something planned?
ivan.borovkov
2018-01-15 12:01
It will recover soon
ivan.borovkov
2018-01-15 12:01
Clean up task
ivan.borovkov
2018-01-15 12:01
It scheduled when no tasks running
ivan.borovkov
2018-01-15 12:02
Doing cleanup and starts jenkins back
ivan.borovkov
2018-01-15 12:03
You can cancel shutdown, i will rework it later today
roman
2018-01-15 13:18
@ivan.borovkov can't restart build https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-2954/
ivan.borovkov
2018-01-15 13:27
It cant merge
adamb
2018-01-15 13:34
There are conflicts that needs to be resolve first
adan
2018-01-15 13:37
@ivan.borovkov is it okay for me to start nightly build now? I can see that 3032 is merged, so build should be pushed to http://artifacts.status.im:8081/artifactory/nightlies-local/ , right?
ivan.borovkov
2018-01-15 13:38
feel free to start it now, yes, you should receive link from artifactory
adan
2018-01-15 13:38
thanks!
ivan.borovkov
2018-01-15 13:39
Idea here is to avoid multiple dependencies. I cant find any other usage of this artifactory except nightly android builds, i think we can use diawi and remove this artifactory completely.
adan
2018-01-15 13:40
we have download limit on diawi, 25 times per build should be
adan
2018-01-15 13:40
which isn't great
adan
2018-01-15 13:41
maybe we should find another way for publishing builds?
roman
2018-01-15 13:44
oh ok
ivan.borovkov
2018-01-15 13:46
But why we have only android builds there? Do we need more than 25 downloads of nightly android build?
adan
2018-01-15 13:48
I think that 1 build can be downloaded several times to different VMs/Devices during testing
adan
2018-01-15 13:49
so we have 6 testers + automated tests which is > 25 downloads per build
adan
2018-01-15 13:50
also we have a lot of links to the nightly storage on wiki pages, e.g. 'You can try one of our nightly builds' etc.
adan
2018-01-15 13:55
I think that it's better to move the topic to core channel since there could be other reasons for using the storage, or maybe there could be some suggestions for using something else
ivan.borovkov
2018-01-15 13:55
we have 239 nightly builds(since 2017-10-06), they were downloaded 31465 times. And 61 builds were downloaded more than 25 times.
agerage downloads per build: 131.653
median: 8
min: 1
max: 11939
adan
2018-01-15 13:56
specific build has been downloaded 11939 times?
adan
2018-01-15 13:56
wow :slightly_smiling_face:
ivan.borovkov
2018-01-15 13:58
builds with 100+ downloads:
``` 100 nightlies-local:im.status.ethereum-ee2904.apk
105 nightlies-local:im.status.ethereum-5206f2.apk
108 nightlies-local:im.status.ethereum-c8168e.apk
109 nightlies-local:im.status.ethereum-0a1c3b.apk
111 nightlies-local:im.status.ethereum-266ffb.apk
114 nightlies-local:im.status.ethereum-38f6db.apk
125 nightlies-local:im.status.ethereum-4022be.apk
139 nightlies-local:im.status.ethereum-2f4c75.apk
141 nightlies-local:im.status.ethereum-19e180.apk
153 nightlies-local:im.status.ethereum-0403a7.apk
162 nightlies-local:im.status.ethereum-279ebc.apk
190 nightlies-local:im.status.ethereum-a7f852.apk
194 nightlies-local:im.status.ethereum-fe78e7.apk
195 nightlies-local:im.status.ethereum-5dd2f8.apk
256 nightlies-local:im.status.ethereum-0f2326.apk
266 nightlies-local:im.status.ethereum-f90e1a.apk
456 nightlies-local:im.status.ethereum-4b6f92.apk
548 nightlies-local:im.status.ethereum-b42df6.apk
559 nightlies-local:im.status.ethereum-3933de.apk
811 nightlies-local:im.status.ethereum-0822e5.apk
979 nightlies-local:im.status.ethereum-07c247.apk
1077 nightlies-local:im.status.ethereum-bcc2b1.apk
2536 nightlies-local:im.status.ethereum-1e9ed8.apk
3356 nightlies-local:im.status.ethereum-7256dd.apk
3510 nightlies-local:im.status.ethereum-024cf6.apk
11939 nightlies-local:im.status.ethereum-ed5973.apk```
ivan.borovkov
2018-01-15 14:04
there seems to be that no other alternatives to diawi
ivan.borovkov
2018-01-15 14:05
but we can upload artifacts for example to s3
adan
2018-01-15 14:15
artifacts on s3 sounds great
adan
2018-01-15 14:16
I would be able to get them via get request, e.g whole list of artifacts or specific artifact by name, right?
anna
2018-01-15 14:42
Just FYI we also share the nightlies with community, so many more people go to artifactory link and download from it, if we are going to change it it should be announced before the change and we need to update info where to get the build on our website, wiki and possibly other sources
chad
2018-01-16 02:37
has joined #devops
chad
2018-01-16 02:51
Hey #devops! I wanted to ping you and let you know about a new service that was spun up to automate onboarding Testflight users. Until now this was time consuming for both users and Status to manage, but now it is instant and was made possible using a Fastlane tool called Boarding. Check it out here https://github.com/fastlane/boarding.
Boarding is a rails app that we are hosting on Heroku, and uses the iTC credentials of a newly created account () with limited permissions. I used Heroku since we could get an MVP up and running in seconds, but am pinging you in case you wanted to run it on our Status servers. Credentials for gmail, iTC account, and Heroku are in lastpass.
The actual signup app can be seen at https://status-boarding.herokuapp.com/, and this will be embedded in an iframe on http://status.im.
ivan.borovkov
2018-01-16 06:29
Hi Chad. Thanks for notifying here. I will try to keep that in mind
oskarth
2018-01-16 07:55
+1 for putting it on S3
oskarth
2018-01-16 08:11
@ivan.borovkov So I'm still a bit out of it but trying to fix versioning info.
1) When I clicked 'build now' for release build it broke within a few seconds. Looking at macos release build it was stuck in some DETACHED 213123 (rand number) state
2) On macos machine I wanted to check some tags stuff (our version info relies on release branches being avail/local). Now git-describe should work properly for versioning data, and it passes step it failed in 1 but
3) Now iOS build fails with "bad revision ^" error. Git looks like it is in normal state (develop)
Any ideas? Guessing it is related to `set_xcode_version.sh` script but no idea why it would start failing now, nor why build generally failed to begin with.
ivan.borovkov
2018-01-16 08:16
We had this discussion with Roman. He said that versioning script is based on number of commits from develop branch. This script is very fragile by itself, and i think it need to be reworked. And it seems to be that ios build cannot go on if there is no new commits from develop (which is normal, when you try to build develop branch).
oskarth
2018-01-16 08:16
Re 1 I also see @anna said:
> 3. We have a mess with app version now in Android it's 13208203 and iOS 0.0.2116 that is very wrong, but it's in TODO list still
But a few days/aweek ago it was:
In last release build, iOS build has `0.9.11d523` version. Will check for exact Android build ver but in general it's something like `Status - 0.9.10-523-gf90e1a4e+`, so iOS has `0.9.11`and Android `0.9.10`
Did something change in this regard?
oskarth
2018-01-16 08:17
> fragile
Would agree with this
ivan.borovkov
2018-01-16 08:17
I did no changes regarding versioning of the builds.
oskarth
2018-01-16 08:18
> BUILD_VERSION=$(git describe --tags --always --dirty=+)
This is what I expect it to be, but there are some problems with tags/branches, or how we use them anyway. Might require some git-fu in script to fix. Now it shows up same on macos as locally anyway `0.9.13-...etc`
oskarth
2018-01-16 08:18
> . And it seems to be that ios build cannot go on if there is no new commits from develop (which is normal, when you try to build develop branch).
Weird, so what is fix for this you think?
oskarth
2018-01-16 08:19
I also forgot we are using normal tags, we should be using annotated tags. And probably not wise to just have them on release branch. Will investigate more for next iteration but I think current tag info state is OK for this release.
oskarth
2018-01-16 08:20
> fatal: bad revision '^'
> Command /bin/sh failed with exit code 128
So are you saying you think this iOS build error error will go away if there's a new commit in develop? Or do you mean release branch?
ivan.borovkov
2018-01-16 08:23
I mean release branch. It should be ahead from develop
oskarth
2018-01-16 08:24
ok
oskarth
2018-01-16 08:25
do you know where this line fails btw? I can't see what triggers it
oskarth
2018-01-16 08:25
maybe we should add echo mode to this xcode script
ivan.borovkov
2018-01-16 08:27
i did some debug recently
ivan.borovkov
2018-01-16 08:27
let me find it
ivan.borovkov
2018-01-16 08:28
here are more details and more correct things
oskarth
2018-01-16 08:29
oh wow
oskarth
2018-01-16 08:29
I see it now, tail n, yeah very brittle indeed, thanks
oskarth
2018-01-16 08:56
@ivan.borovkov Thoughts on where/how we can host this? https://github.com/status-im/status-github-bot/issues/1#issuecomment-357881750
@pedro Didn't test it yet, but approach seems reasonable to me :slightly_smiling_face:
cc @yenda
ivan.borovkov
2018-01-16 08:59
at this moment we have no centralised platform for hosting apps
ivan.borovkov
2018-01-16 09:00
if we have AWS account we can try this:
https://aws.amazon.com/about-aws/whats-new/2018/01/aws-lambda-supports-go/
ivan.borovkov
2018-01-16 09:00
If we dont have AWS account - we can create one and then try lambda go :slightly_smiling_face:
adan
2018-01-16 09:46
@ivan.borovkov 502 Bad Gateway on the Jenkins.
oskarth
2018-01-16 09:55
eta on it being back?
ivan.borovkov
2018-01-16 09:57
should be back now
ivan.borovkov
2018-01-16 09:57
but need to ensure that it is running correctly
oskarth
2018-01-16 09:57
ah yes indeed, thanks
oskarth
2018-01-16 09:57
can I use it?
ivan.borovkov
2018-01-16 09:58
i?ve moved it from userspace to docker container
ivan.borovkov
2018-01-16 09:58
yes, feel free please
oskarth
2018-01-16 09:58
ok
oskarth
2018-01-16 09:58
I just pushed a tag to release
oskarth
2018-01-16 09:58
hopefully it picks it up
ivan.borovkov
2018-01-16 09:59
all pushes that were done during jenkins offline were not received
ivan.borovkov
2018-01-16 09:59
webhooks need to be redelivered for this
oskarth
2018-01-16 09:59
ok, I triggered build
oskarth
2018-01-16 09:59
also not sure if tags count same as commits so might fail anyway
oskarth
2018-01-16 10:02
hm nope
oskarth
2018-01-16 10:22
@ivan.borovkov https://jenkins.status.im/job/status-react/job/release/job/release%252F0.9.13/58/console docker port issue maybe?
oskarth
2018-01-16 10:22
```
java.lang.SecurityException: Rejected: org.jfrog.build.api.Artifact
at hudson.remoting.ClassFilter.check(ClassFilter.java:75)
```
ivan.borovkov
2018-01-16 10:25
ok, ill revert it, lets see
oskarth
2018-01-16 10:26
Also re stuff like `error: pathspec 'develop' did not match any file(s) known to git.` Seems like it is quite gnarly with all the Jenkins plugin stuff https://stackoverflow.com/questions/45399894/is-it-impossible-to-checkout-a-different-branch-in-jenkinsfile?rq=1
oskarth
2018-01-16 10:27
I think any job should work just by clicking "build now" but it isn't obvious to me how to make this work for release build wrt this error (develop shows up under `git show-ref` on machine, but might be a different job or smt, it's in elease-react...folder)
I'll try with this `git checkout -t -b develop origin/develop` but not sure if it'll help, lets see
ivan.borovkov
2018-01-16 10:30
i?ve reverted jenkins to previous version, you can try now
pedro
2018-01-16 10:30
thanks @oskarth :thumbsup: If it sounds reasonable, I don't mind quickly implementing a couple of other use cases from the wishlist, just so we can evaluate the time required, and how that affects the codebase. That way we'll get a better sense of whether we want to go in this direction or not.
oskarth
2018-01-16 10:30
ok thanks
oskarth
2018-01-16 10:30
@pedro sounds good
oskarth
2018-01-16 10:30
any other automation stuff you can think of would also be good, wishlist just a start with some stuff
oskarth
2018-01-16 10:31
anything that people do manually is of course up for automation
pedro
2018-01-16 10:31
Eventually we might want to bring stuff like `merge-external-pr.sh` to Slack, right?
oskarth
2018-01-16 10:32
possibly yeah
oskarth
2018-01-16 10:32
I'd like if we could print daily TODO lists or so for people in slack
pedro
2018-01-16 10:32
I can see an advantage, which is that once you kick it off in Slack, you could for instance turn off your laptop and finish the flow on your phone, if you're on the go
oskarth
2018-01-16 10:32
so we evenly distribute review requests etc, and then people can be notified about state at some interval
oskarth
2018-01-16 10:33
yeah possibly
oskarth
2018-01-16 10:33
need to figure out how we want to deal with signing and also if something goes wrong
oskarth
2018-01-16 10:33
but why not
oskarth
2018-01-16 10:33
good idea
oskarth
2018-01-16 10:34
it'd also be cool to gamify it a bit
oskarth
2018-01-16 10:34
so there's a point system for reviews, and possibly for regressions not caught in review too
oskarth
2018-01-16 10:34
similar to bug fixing days etc, making lightweight reputation/competition systems
pedro
2018-01-16 10:35
sounds cool!
oskarth
2018-01-16 13:59
awesome @chad, sorry I spaced out on keeping up with slack a bit
chad
2018-01-16 13:59
Np! Hope you are feeling better!
oskarth
2018-01-16 14:00
worst part over, I just had food! after some sleep I'll be back to normal hopefully :slightly_smiling_face:
oskarth
2018-01-16 14:00
feel free to review @pedro!
chad
2018-01-16 14:03
Will you eat oysters again?
oskarth
2018-01-16 14:07
of course, back on the bike!
oskarth
2018-01-16 14:07
I'll skip the fugu this time though http://www.bbc.com/news/world-asia-42699600
hutch
2018-01-16 14:39
has joined #devops
chad
2018-01-16 15:02
Fastlane is pretty sweet @ivan.borovkov. Have you taken a look at how some other companies integrate it into their workflow? https://github.com/fastlane/examples/blob/master/README.md
adan
2018-01-16 18:34
@ivan.borovkov did you remove http://artifacts.status.im:8081/artifactory/nightlies-local/ ? I can't access it, long loading and "This site can't be reached" in ~40 seconds
ivan.borovkov
2018-01-16 18:34
i did nothing with artifactory
chad
2018-01-16 18:37
This one looked interesting https://probot.github.io/apps/stale/ (30% of our status-react issues are over 6 months old)
adan
2018-01-16 18:43
@ivan.borovkov Please take a look when you can, the artifactory can't be reached from my side, the same behavior has @anna
chad
2018-01-16 18:44
> Heads up! Note that most apps in the @probot organization have an official hosted app that you can use for your open source project. Use the hosted instance if you don't want to deploy your own.
ivan.borovkov
2018-01-16 18:45
check now?
adan
2018-01-16 18:45
It works!
adan
2018-01-16 18:45
Thanks!
pedro
2018-01-16 21:22
@chad Probot looks really slick! Thanks for the tip.
oskarth
2018-01-17 05:44
> Jenkins is going to shut down
ivan.borovkov
2018-01-17 06:41
Funny.
There is macos1 cleanup job that is started at midnight, then it puts jenkins into shutdown mode, waits for other tasks to be finished, doing cleanup and then cancels jenkins shutdown. But seems to be that something went wrong :slightly_smiling_face:
oskarth
2018-01-17 10:20
https://github.com/status-im/ideas/issues/69 fyi @ivan.borovkov think this one needs your input
oskarth
2018-01-17 10:20
maybe it waiting for itself to be finish? :x
ivan.borovkov
2018-01-17 10:21
nop, it worked previous night, and didnt work this, not sure about real reason for this. I changed its settings, should not block work anymore
oskarth
2018-01-17 10:22
yeah wasn't blocking, just noticed it
dmitry.shulyak
2018-01-17 10:22
has joined #devops
ivan.borovkov
2018-01-17 10:23
yeah, i saw this. Currently there are much more fundamental problems that need to be solved before thinking about new cluster, but eventually this will be done, yes.
ivan.borovkov
2018-01-17 10:23
It is not worth to have any assumptions for now, as they may change later
oskarth
2018-01-17 10:23
> Currently there are much more fundamental problems that need to be solved before thinking about new cluster,
agree
oskarth
2018-01-17 10:24
yeah think it is fair, you might want to comment on this so it is in line with your vision of infra, and at what point you think a decision should be made
oskarth
2018-01-17 10:24
so swarm can still make progress with current infra and fundamental problems are solved etc
oskarth
2018-01-17 10:25
assuming they'll start soon, I guess it isn't unlikely it'll be delayed
ivan.borovkov
2018-01-17 10:26
There is not so much to do with this for devs
ivan.borovkov
2018-01-17 10:26
at least at this stage
oskarth
2018-01-17 10:27
so I'm not right person to talk to, just nudge to chat with @adamb and everyone else in that thread :slightly_smiling_face:
oskarth
2018-01-17 10:27
he posted idea so
pedro
2018-01-17 11:07
Looks like https://jenkins.status.im/job/status-react/job/parametrized_build/330/ has been running for almost 2h
oskarth
2018-01-17 11:11
probably easiest to cancel and retrigger
ivan.borovkov
2018-01-17 12:06
This jobs were hard stuck, i had to restart jenkins server to unstuck them. Queue is processing now in normal mode
chad
2018-01-17 22:19
@ivan.borovkov FYI I forked the Boarding repo and it now lives here https://github.com/status-im/boarding
chad
2018-01-17 22:22
Any new commits here will need to be manually deployed to Heroku by running `git push heroku master` once Heroku is setup locally (auto deployment can be setup too).
yenda
2018-01-17 22:31
@ivan.borovkov I think there is no space left on device again https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-3027/6/console
ivan.borovkov
2018-01-18 07:13
There is macos cleanup job, feel free to run it when you face problem with no space
adan
2018-01-19 12:47
why are we publishing release builds to http://artifacts.status.im:8081/artifactory/nightlies-local/ ?
adan
2018-01-19 12:48
Right now I have 2 branches with 2 different states of e2e tests, one branch for nightlies (with new design) another branch for release (old design)
adan
2018-01-19 12:50
so when tests are running against latest nightly build, tests are getting release build
adan
2018-01-19 12:52
@ivan.borovkov ^
ivan.borovkov
2018-01-19 12:53
This happens because jenkinsfile for releases says to upload artifact to artifactory
adan
2018-01-19 12:54
okay, do we need the release build on artifactory or we can publish them on diawi?
ivan.borovkov
2018-01-19 12:56
It is better to ask testteam about this
adan
2018-01-19 12:57
My understanding of 'nightlies-local' - build which should been built each night from develop branch
adan
2018-01-19 12:57
but maybe I am missing something
adan
2018-01-19 13:00
testteam do we need release builds in nightly storage? ^
ivan.borovkov
2018-01-19 13:46
Most probably you are concerned about this lines: https://github.com/status-im/status-react/blob/develop/Jenkinsfile.release#L72-L74
feel free to create pull requests that will put release into separate target.
pedro
2018-01-19 18:38
: If someone could install https://github.com/apps/status-github-bot on the status-react repo, I'd appreciate it. It'll start taking care of some basic PR maintenance
dmitryn
2018-01-23 14:12
@ivan.borovkov could you help to understand why this is failing https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-3110/#
we need that to fix release branch bug
ivan.borovkov
2018-01-23 14:25
There was never mentioned about workflow when you try to create pull request to release branch. Usually someone just picking PR?s from develop and push them to release.
dmitryn
2018-01-23 14:36
Ok. We also have issue when parametrized build based on PR branched off release branch could upload wrong apk for android (ios artifact is ok)
Example:
https://jenkins.status.im/job/status-react/job/parametrized_build/352
branch `bug/npe-#3081`
dmitryn
2018-01-23 14:37
Can we use parametrized build to build PR made to release branch?
ivan.borovkov
2018-01-23 14:39
There is a fix in develop that fixes apk upload, rebase to develop.
dmitryn
2018-01-23 14:46
@ivan.borovkov thanks, i hope that was the fix i cherry-picked https://github.com/status-im/status-react/commit/0eb4fb0a4234ed5636794480c29f04d9859a97dc (cannot rebase because branch based on release)
ivan.borovkov
2018-01-23 14:48
yes, looks like this
tatiana
2018-01-24 14:00
Hi all! I want to ask install https://wiki.jenkins.io/display/JENKINS/Config+File+Provider+Plugin for SOB autotests, because I cannot manage using secretfile as config for autotests folder. Apparently it is because of lack pipeline syntax knowledge and .
What should I do to make everything work:
- copy particular file from jenkins credentials (it is already placed in Jenkins Credentials as 'config.ini')
- paste it in particular place to repo
If anybody can help me with pipeline script for https://jenkins.status.im/job/end-to-end-tests/job/sob-end-to-end-tests/ and current Jenkins Credentials implementation (I tried to modify it using `withCredentials([file(credentialsId: 'config.ini', variable: 'CONFIGI')]) {
'use $CONFIGI` in `try` loop - but I didn't succeed) - highly appreciated!
So at the moment I believe only separate plugin is simplest decision.
@ivan.borovkov @adan what do you think about it?
adan
2018-01-24 14:01
I can try
ivan.borovkov
2018-01-24 14:01
`Config File Provider Plugin` is already installed on jenkins
adan
2018-01-24 14:02
If plugin is easier to use and you can do that, I won't try anything :slightly_smiling_face:
tatiana
2018-01-24 14:02
@ivan.borovkov hm, is it enabled?
ivan.borovkov
2018-01-24 14:03
It is enabled
tatiana
2018-01-24 14:06
@adan hold on, I try to figure out how to use `Config File Provider Plugin` on jenkins and let you know about further details
adan
2018-01-24 14:10
Also you can use manual configuration with values in dropdowns and input fields, instead of pipeline. If you are not comfortable with the pipeline.
But the best practice is to use pipeline from Jenkinsfile. Since actual pipeline is not in the file, yet, you can create one :slightly_smiling_face:
anna
2018-01-29 10:32
@ivan.borovkov any known issues on jenkins now? I?m trying to build https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-3137/5/console and have ``` /Users/jenkins/workspace/react_pull_requests_PR-3137-GNBACVYXYA35RVMIJDKZMAHPU52ATT2ITXLQCV3NIP2Z5I56WEPA/node_modules/instabug-reactnative/ios/RNInstabug/RCTConvert+InstabugEnums.m:10:9: fatal error: 'Instabug/IBGTypes.h' file not found
#import <Instabug/IBGTypes.h>
^~~~~~~~~~~~~~~~~~~~~
1 error generated. ```
ivan.borovkov
2018-01-29 10:33
no known inssues. Probably some versions changed in dependencies or something like this, dev should be able to find reason of this error and fix.
anna
2018-01-29 10:33
cc @andrey
alwx
2018-01-29 10:39
have the same problem with my PR
alwx
2018-01-29 10:40
ok, it's not exactly the same, but looks similar
```
1 error generated.
/Users/jenkins/workspace/react_pull_requests_PR-2795-L26DQEO5RMY6AH7YRPAGG7IFOXRMYCZHD677EXS2CRNATRS2SQBQ/modules/react-native-status/ios/RCTStatus/RCTStatus.m:6:9: fatal error: could not build module 'Instabug'
@import Instabug;
~~~~~~~^~~~~~~~
** ARCHIVE FAILED **```
ivan.borovkov
2018-01-29 12:35
There was a problem in global CocoaPods caches, i?ve cleaned cache, it should build now @alwx
alwx
2018-01-29 12:38
cool, thanks @ivan.borovkov
oskarth
2018-02-05 06:09
@ivan.borovkov http://jenkins.status.im seems down to me when I'm logged in to google
ivan.borovkov
2018-02-05 06:23
works for me
alwx
2018-02-05 16:38
hm guys
https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-3173/
it throws an error that tests were not passed and points to i18n file. but there are no changes to i18n now!
(they were done before in this branch, but no I reverted all them back ? maybe it's related somehow?)
alwx
2018-02-05 16:42
ok nvm, greek translation is wrong
igor680
2018-02-06 10:04
@ivan.borovkov Is it possible to setup a custom path to `Jenkinsfile` for `status-go`? We?d like to move it away from the root folder.
ivan.borovkov
2018-02-06 10:05
Yes, sure
ivan.borovkov
2018-02-06 10:05
It could be set up in jenkins job
igor680
2018-02-06 10:05
Gotcha! Thanks!
ivan.borovkov
2018-02-06 10:06
(and actually this script is not used at this moment)
ivan.borovkov
2018-02-06 10:06
so after change please change setting to use jenkinsfile from scm and input correct path :slightly_smiling_face:
ivan.borovkov
2018-02-06 10:06
or let me know i will help you with this
anna
2018-02-08 10:00
^ @ivan.borovkov do we plan any maintenance/restart?
ivan.borovkov
2018-02-08 10:12
@anna should be fine now
adamb
2018-02-08 10:18
@ivan.borovkov all executors failed to connect after the restart :disappointed:
ivan.borovkov
2018-02-08 10:19
they will connect eventually
ivan.borovkov
2018-02-08 10:19
this is because they are not jenkins executors, and servers are managed by ssh
ivan.borovkov
2018-02-08 10:19
it happens always after restart
ivan.borovkov
2018-02-08 10:20
ah, i see, will check now
adamb
2018-02-08 10:21
interesting why it happened after restart. It looks like know host keys got wiped out
ivan.borovkov
2018-02-08 10:53
Fixed, and all agents are live
ivan.borovkov
2018-02-08 10:53
@adamb i?ve restarted your build
adamb
2018-02-08 11:06
Thank you @ivan.borovkov!
kim196
2018-02-12 14:02
Hoping to get some help with faucet- did request a couple days ago and got confirmation of request received but nothing after that...
ivan.borovkov
2018-02-12 14:03
Hi. What is destination wallet?
kim196
2018-02-12 14:04
My ETH wallet- do you need the code?
ivan.borovkov
2018-02-12 14:05
I need wallet that you used in faucet form
kim196
2018-02-12 14:15
@ivan.borovkov is this what you need?
ivan.borovkov
2018-02-12 14:15
i need wallet id, do you have it ?
kim196
2018-02-12 14:21
@ivan.borovkov 0x8a82e5d294abc80ff227eda87f7282126b671a7f is this it? I don't see any other wallet info
ivan.borovkov
2018-02-12 14:30
in faucet service i?ve found this line: `{"hash":"0x2f65e8fae01778ae3cd34b73117b6ae55533f612167ef012d0aa272d477d0193","nonce":"0x1639","gasPrice":"0x3b9aca00","gas":"0x15f90","to":"0x8a82e5d294abc80ff227eda87f7282126b671a7f","value":"0x16345785d8a0000","input":"0x307830","v":"0x29","r":"0xb5acda0beca8e3b0fcc747f0c39a59bf672a0fa575a4ec76f1d76ca9ab38e64","s":"0x217f6a58be891ce0a8fccc88ddb3464556d2b3db290cc02d4099319dbecf4fec"}`
no logs regarding this wallet in geth service.
Transaction seems to be done, probably something wrong anywhere else.
pedro
2018-02-13 09:13
: who would be able to help me set up a new mailserver and bootnode cluster (dedicated to Whisper v6)?
adamb
2018-02-13 09:19
I can help with mailserver, but probably we need a new box to do that. The current one I use has already two MailServers and Prometheus and it's slow
adamb
2018-02-13 09:20
@ivan.borovkov can you help with a new machine in scaleway?
pedro
2018-02-13 09:20
Yeah, if the goal here is to test a performance improvement, we definitely want to start in a good spot
ivan.borovkov
2018-02-13 09:20
Sure, what size is needed?
adamb
2018-02-13 09:23
I guess we?d like to run there two nodes but only running Whisper V6. In theory, 2GB and 50GB SSD should be enough. @pedro is that correct?
pedro
2018-02-13 09:24
I believe so, but maybe @jeka has more insight on that
ivan.borovkov
2018-02-13 09:25
ubuntu 17.10 lts is good?
jeka
2018-02-13 09:26
@adamb you've well described our requirements.
ivan.borovkov
2018-02-13 09:28
please drop me your ssh public key
ivan.borovkov
2018-02-13 09:29
163.172.168.202
51.15.194.39
pedro
2018-02-13 09:29
```
ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQDn+8Nii9BYEqoPFBanH0A5yPBSa1QL43lMdhSQI8tDKbpqoec9syQTcgiFgs2r9++4m7Ni+fsxx7KCUEG5icoT0WLyokt+dazfzs/Y8D4Fsazk/lZVnk+wb3FFebV94tXPYY3w7pxnlWRHHDRzyFjeGVr4l28utvykFLMylBw7873RTjrMwiu+LdZxnoaEkUwVEEsrD23TWDG887OoqyDsUhL/Kj3kzaDxJBFOqakyblZbasIVyR/iVRvAArKgt5c70reWonlk06hhaLoDJdjAKNi4s0Ov4h0HCMj/SweQrLLfeAr1BKA+iH4N4JoDRCpSl8bWnnARkPdkzZRNcOJh6BeT0uQAJ9GO+/ey4P3f05nzAUe8Tfhvs6t9ymCQ/IRGzUaTqGSFVY8qsGe95OcTrKC6xVGaEroR3sFqAn3tLD0hC9Gwgf6qESIxX/DtEDC3GrNQetSJKwsxQc5VuwknW1e4z9GZgY8BwKmOF7T/qcnOG9lruSY2A6maeyEVcCAJYrcgofxefqKRH6im4n1TqCzlbQKuooOz3S51th34I0GN0u+mQW7jqKX0ZDBjjzNRoQPQ+Cmb8tLhoHcOmzn3NPu+Uxtv7pqCF2PpQnnWF3uuTdtIUBJGEIGTMMp+ptK37CnaYdR1M6T14UYro7J8QOMRjqfVVUDhgpTrwNeQ1Q== 54:3b:f3:89:89:d5:01:5b:eb:db:b7:1c:43:b5:3f:dd http://home.pedropombeiro.com
```
ivan.borovkov
2018-02-13 09:30
added.
ivan.borovkov
2018-02-13 09:31
My laptop&phone charger is dead, i?m going to find replacement now, will be not available for some time
roman
2018-02-13 10:41
it seems that this link is public https://jenkins.status.im/job/status-react/job/release/configure , is this expected?
roman
2018-02-13 10:42
oh nice i can't change setting, then nvm
cryptowanderer
2018-02-13 10:43
I can see that page and change settings I think? Well, I tried editing the description just to test, clicked `Save` and have been redirected here https://jenkins.status.im/job/status-react/job/release/indexing/console without logging in or anything
cryptowanderer
2018-02-13 10:44
surely that's not good?!
roman
2018-02-13 10:44
oh, so it actually worked for you?
cryptowanderer
2018-02-13 10:45
yeah, I can see credentials and everything now without logging in :scream:
roman
2018-02-13 10:45
that's not good then
roman
2018-02-13 10:45
@ivan.borovkov
roman
2018-02-13 10:46
i can edit description but can't save it
roman
2018-02-13 10:46
@cryptowanderer are you sure you are not logged in?
roman
2018-02-13 10:46
oh fuck i see
roman
2018-02-13 10:47
jeez, that's bad
cryptowanderer
2018-02-13 10:47
yip :thisisfine:
ivan.borovkov
2018-02-13 10:50
Thats unexpected. Will take a look
ivan.borovkov
2018-02-13 10:51
Should be fixed
cryptowanderer
2018-02-13 10:51
nice! Awesome work @ivan.borovkov
roman
2018-02-13 10:54
yep, it's fixed, thanks
ivan.borovkov
2018-02-13 10:57
. I?m stuck in the middle of nowhere with dead laptop charger and no possibility to buy replacement. Will be trying to evacuate asap from here back to charger. May take few days.
pedro
2018-02-13 13:11
@ivan.borovkov when you are back in the game, please let me know so we can see why I still can't get into those two servers. I try to log in as root and it asks me for the password
ivan.borovkov
2018-02-13 13:11
I have kindof 30% of battery for urgent cases
ivan.borovkov
2018-02-13 13:11
ill check now
ivan.borovkov
2018-02-13 13:12
check now please
pedro
2018-02-13 13:18
now it worked, thanks!
adan
2018-02-14 09:52
Build for Android is crashing randomly due to `Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)`
e.g.
https://jenkins.status.im/blue/organizations/jenkins/status-react%2Fpull%20requests/detail/PR-3311/3/pipeline
@ivan.borovkov is it possible to fix that?
ivan.borovkov
2018-02-14 10:31
Yes, by adding one more macos machine or by adding more resorces to this.
igor680
2018-02-14 17:49
`status-go-manual` job is stuck with `Waiting for next available executor on linux` for over an hour already.
tatiana
2018-02-14 18:05
I stopped all open-bounty builds, so `status-go-manual` job should be built in a short time
igor680
2018-02-14 18:21
Thanks!
dmitryn
2018-02-15 12:31
@ivan.borovkov can we use Artifactory to store custom artifacts? We use ?re-frame? library as dependency in status-react, but switching to our own fork https://github.com/status-im/re-frame/tree/0.10.2 Now i?m using http://clojars.org under my personal account https://clojars.org/dmitryn/re-frame-fork, but it should be changed.
ivan.borovkov
2018-02-15 12:32
yes, sure
dmitryn
2018-02-15 12:32
where are the credentials?
ivan.borovkov
2018-02-15 12:36
They are already built into jenkins, so you can check code of status-react nightly and change to your needs.
ivan.borovkov
2018-02-15 14:56
i?m not using it to track tasks/activity
naghdy
2018-02-15 14:59
No problemo. Whats the best place to track activity? Can we close down the project then?
ivan.borovkov
2018-02-15 14:59
Yes, we can close project. Most of activity is tracked here in slack
oskarth
2018-02-15 15:00
sorry, but isn't it better to use the project than doing it ad hoc in slack? Slack is not a great tool for tracking tasks :slightly_smiling_face:
oskarth
2018-02-15 15:02
(or some other solution that doesn't get lost / is easy to reference)
ivan.borovkov
2018-02-15 15:04
It most probably would not display full state of things in devops. To make it right we will need to route all activity to issues, because i get a lot of small requests during day. Not sure that at this moment this is required overhead for interaction.
oskarth
2018-02-15 15:05
I think the same is true for any dev, designer, PM or QA re small requests. What about tracking big pieces of work?
ivan.borovkov
2018-02-15 15:06
I?m trying to keep most important things there, and once per some period of time doing updates/cleanup
oskarth
2018-02-15 15:06
sounds good! so the project is still needed then? or is "there" somewhere else?
ivan.borovkov
2018-02-15 15:07
And i think that it need to be done more often, will try to do it
ivan.borovkov
2018-02-15 15:07
i dont think that project is needed, i?m tracking epics in status-cluster issues
oskarth
2018-02-15 15:07
you mean like regular GH issues and labels or something?
oskarth
2018-02-15 15:07
ok sounds good
oskarth
2018-02-15 15:08
not quite sure where to look
ivan.borovkov
2018-02-15 15:08
yes
deodatus
2018-02-18 20:29
I guess this should appropriately be here. Apologies if not :pray:
Someone asked on Riot about testflight, said he never got one even after registering already. So i told him to re-register and thoroughly check his spam folder. Dm'ed me said not getting any even after registering again. So could anyone with access to list of testflight users check if this email already in it, if its not, maybe manually send an invite please? Ty!
ivan.borovkov
2018-02-18 20:56
@cryptowanderer can you please help with this?
chad
2018-02-18 20:59
I can help. What?s his email?
deodatus
2018-02-19 09:32
His email is
jeluard
2018-02-21 08:58
Looks like jenkins build are triggered when a PR is merged. Maybe we could get rid of those?
ivan.borovkov
2018-02-21 09:10
Hi. Build of what is triggered?
jeluard
2018-02-21 09:13
PRs
jeluard
2018-02-21 09:13
jeluard
2018-02-21 09:14
I cancelled 2 and 3 that were triggered by closing associated PR
ivan.borovkov
2018-02-21 09:17
I updated webhook, let me know if it helped
jeluard
2018-02-21 09:17
:+1:
jeluard
2018-02-21 09:17
Thanks!
jeluard
2018-02-21 09:22
Is someone looking at having diawi links posted to GH as part of jenkins build process?
jeluard
2018-02-21 09:25
BTW new jenkins UI is awesome!
jeluard
2018-02-21 09:25
With QR code even better
ivan.borovkov
2018-02-21 10:13
It should be relatively simple using this:
https://stackoverflow.com/questions/46382950/how-send-a-comment-in-a-github-pullrequest-from-jenkins-jenkinsfile
dmitryn
2018-02-21 10:34
Jenkins down? Not building anything on last commit here https://github.com/status-im/status-react/pull/3332
dmitryn
2018-02-21 10:36
I will Run it manually then
ivan.borovkov
2018-02-21 10:47
PR was last updated 18 hours ago, and last build was done 18 hours ago. Not sure what do you mean on `jenkins down` and `not building anything`
dmitryn
2018-02-21 10:50
@ivan.borovkov PR was updated 28 mins ago and jenkins didn't start the build. I have to run it manually from dashboard
ivan.borovkov
2018-02-21 10:57
Ok, most probably it is related to this change:
https://status-im.slack.com/archives/C8M1Z25J8/p1519203482000186
I've reverted this settings, as it is causing confusion from other team members @jeluard
jeluard
2018-02-21 10:59
What was the setting? The only one that shouldn't be triggered is after the PR is merged
dmitryn
2018-02-21 11:00
Every commit made to PR should trigger build
jeluard
2018-02-21 11:02
When merging a build is triggered, this is the one that could be avoided
ivan.borovkov
2018-02-21 11:06
GH isnt very flexible on this configuration:
https://s.esteq.net/n7ycsqy7bw.png
jeluard
2018-02-21 11:08
:+1:
dmitryn
2018-02-21 13:36
@ivan.borovkov are you going to return auto builds on new commits setting back?
ivan.borovkov
2018-02-21 13:37
It should be back.
dmitryn
2018-02-21 13:38
i just pushed new commit to https://github.com/status-im/status-react/pull/3225
dmitryn
2018-02-21 13:38
And build was not started
dmitryn
2018-02-21 13:39
but it was force push, not new commit
ivan.borovkov
2018-02-21 13:39
```continuous-integration/jenkins/pr-merge ? This commit is being built```
dmitryn
2018-02-21 13:40
Thanks, now it's building - maybe there was a lag
jeluard
2018-02-22 14:55
What does it mean when all builds are waiting for `macos1` ? ,https://jenkins.status.im/blue/pipelines/
ivan.borovkov
2018-02-22 14:56
This machine has max 2 builds running at the same time, both already building, so others are set in queue
jeluard
2018-02-22 14:57
Where can I see running ones?
jeluard
2018-02-22 14:58
Can't see them with the new interface?
ivan.borovkov
2018-02-22 14:59
i dont see them there
jeluard
2018-02-22 15:00
Also is that expected that Android build is running on macos while linux build are idle?
ivan.borovkov
2018-02-22 15:00
in jenkinsfile it is bind to macos1, so it cant be different
jeluard
2018-02-22 15:01
hmm any reason to do that?
ivan.borovkov
2018-02-22 15:06
probably it was much more complicated to have them run distributed.
igor680
2018-03-12 13:55
these `linux1`, `linux2` machines, do they have Docker and Docker-compose installed?
tatiana
2018-03-12 14:15
they definitely have `docker`, and don't know about Docker-compose
igor680
2018-03-12 14:15
@tatiana can you run `docker-compose version` there to see if they have? :slightly_smiling_face:
tatiana
2018-03-12 14:19
I don't have access, sorry
tatiana
2018-03-12 14:20
I know only because I launched tests through Jenkins and docker recently
igor680
2018-03-12 14:20
hmm, I have an idea then?
igor680
2018-03-12 14:24
thanks!
```
+ docker-compose version
/tmp/jenkins6082815865929441517.sh: 3: /tmp/jenkins6082815865929441517.sh: docker-compose: not found
Build step 'Execute shell' marked build as failure
Finished: FAILURE
```
:slightly_smiling_face:
igor680
2018-03-12 14:24
no compose then
igor680
2018-03-13 17:21
I?m trying to add a new node to Jenkins for my tests. Can I get a public ssh key to add it to node?s `authorized_keys`?
pedro
2018-03-13 18:04
Not sure I understand your question. You want to use `ssh-keygen` to generate a key that you can add to `authorized_keys`, right? https://www.digitalocean.com/community/tutorials/how-to-set-up-ssh-keys--2
igor680
2018-03-13 18:07
I?m trying to add a new runner node for Jenkins.
It?s init command is `ssh -v <ip-of-the-node> agent`
That ssh connection is from Jenkins host ? my droplet
I don't have access to the Jenkins host machine :slightly_smiling_face:
pedro
2018-03-13 18:08
ah, got it :slightly_smiling_face:
igor680
2018-03-13 18:09
So I thought, mb somebody knows Jenkins? public SSH key or has access to this machine :slightly_smiling_face:
roman
2018-03-19 08:22
@adamb ^
roman
2018-03-19 08:22
if this is still relevant though
adamb
2018-03-19 08:24
It has been handled
roman
2018-03-19 08:25
thanks, sorry i first responded then figured that thread is a bit old :slightly_smiling_face:
adamb
2018-03-19 08:33
No worries, it's good you checked it :+1:
martin487
2018-03-21 11:35
Spreadsheet with all servers: https://docs.google.com/spreadsheets/d/1By53othxtxtMOS01Wv13W99cvKlmWuKP35ATHQjtJ4w/edit#gid=0
martin487
2018-03-21 11:35
@martin487 pinned a message to this channel.
chad
2018-03-21 11:36
I know we have at least 2 Heroku servers too
jacqueswww
2018-03-21 11:37
Haha
jacqueswww
2018-03-21 11:37
@chad please pop the details in. if you could/know them ?
chad
2018-03-21 11:40
Done. I think @pedro had the other?
igor680
2018-03-21 11:44
I have 2 servers in my own DO account that are used, one for providing `#humans-need-not-apply` messages and having a web UI for `#status` chatrooms and another is for idea 91 prototyping.
jacqueswww
2018-03-21 11:45
If you want to recreate them on Status DO, let me know and I will create the instances for you
jacqueswww
2018-03-21 11:45
Thanks @chad :heavy_check_mark:
igor680
2018-03-21 11:46
yes, I think I'll do that eventually, I'll use mine to figure out the exact requirements :slightly_smiling_face:
martin487
2018-03-21 11:47
@jacqueswww do you have access to Status owned DO account? If I wanted to could I get an API token from you?
jacqueswww
2018-03-21 11:48
sure
pedro
2018-03-21 12:43
Added status-github-bot Heroku instance
naghdy
2018-03-21 19:30
Thanks for setting this up @martin487. It makes onboarding devops much easier now.
martin487
2018-03-21 19:48
I think @jacqueswww set this up,I just.posted the link :joy:
jacqueswww
2018-03-22 07:08
@naghdy @martin487 Please share around to other people that have more parts of the server puzzle :wink:
chad
2018-03-22 07:10
@pedro we should merge those 2 heroku servers into one account for easier managing. ping me when you are around today
chad
2018-03-28 05:35
:coffee: Morning!
tl:dr Can someone please help me setup a Jenkins job to run nightly?
chad
2018-03-28 05:37
We have a recurring problem where we max out our 10k Testflight users which breaks alpha signup. To fix this we need to manually run a script that removes inactive testers. Problem is that usually this happens *after* users start seeing signup errors due to the 10k user limit being maxed.
chad
2018-03-28 05:37
So running the script daily should keep the issue in check.
dmitryn
2018-03-28 08:25
We have `Jenkins.nightly` file which makes nightly builds. You could add any script call there by using `sh ('/path/to/script')`. For example, Fastlane script https://github.com/status-im/status-react/blob/develop/Jenkinsfile.nightly#L84
Maybe, it makes sense to create new `stage` for that matter
arnetheduck.slack
2018-03-28 08:39
hey, who runs this place? :wink: could use some help to set up jenkins servers for #67-nimbus for a bunch of targets (x86, arm, matrix of operating systems)
adamb
2018-03-28 08:42
Hey, no one really, we are still in progress of hiring a devops person. Does it have special requirements? Travis or Circle CI is not enough?
arnetheduck.slack
2018-03-28 08:46
likely not - we need arm/x86 support, as well as windows/mac/linux/ios/android
adamb
2018-03-28 08:54
And you need all these as native architectures? :open_mouth: Currently in Jenkins we have only linux x64 and macos. On Scaleway I saw ARM but they were out of stock at the time.
arnetheduck.slack
2018-03-28 08:55
that's the nature of the beast
adamb
2018-03-28 09:01
I can help with writing a Jenkinsfile to run a job on the current executors but with the rest you guys need to find a place where these architectures are available. Then we can connect them to our Jenkins master (ssh access is enough + Java installed on those machines). macOS should be enough to build iOS and linux to build Android libraries.
oskarth
2018-04-03 03:34
@oskarth set the channel topic: Servers: https://docs.google.com/spreadsheets/d/1By53othxtxtMOS01Wv13W99cvKlmWuKP35ATHQjtJ4w/edit#gid=0
martin487
2018-04-05 15:06
are we currently/planning to operate IPFS nodes/gateway?
martin487
2018-04-05 15:07
We need a blob-store for https://github.com/status-im/open-bounty/issues/384#issuecomment-378967721 and obviously it would be nice to use IPFS instead of S3 or similar
cryptowanderer
2018-04-06 06:49
Ideally, I would think we should, but it might be tough without a dedicated devops person. I have worked a little with ipfs before and can help set it up and run a node for you, but I'm not convinced it will be the most robust solution...
martin487
2018-04-06 07:28
@cryptowanderer we might consider infura and switch to status operated gateway as soon as there is one
andreaf
2018-04-06 08:25
we are currently playing with swarm instead of IPFS for #084-restore-account
jeka
2018-04-10 08:57
Just curious, is there any progress with eth swarm?
hutch
2018-04-10 13:58
did you guys see this? https://press.swarm.city/ipfs-consortium-release-6907f0b55bbf
hutch
2018-04-10 13:58
@martin487 ^
hutch
2018-04-10 14:01
Also this - https://bra.twurst.com/articles/eth-loves-libp2p.html (older article but super interesting, if anyone was at / around / involved in devcon probably already seen but yeah :slightly_smiling_face: )
oskarth
2018-04-16 11:14
@adamb @arnetheduck.slack what happened with this? we have devops starting next Monday so this might be a nice greenfield-ish project
oskarth
2018-04-16 11:15
@pedro @adamb Re status-cluster: anything that needs to be done on devops side here next month or so that you can think of?
adamb
2018-04-16 11:18
There is a lot of stuff to do there. Nothing that we can't do by ourselves but it would be better to have someone who can dedicate their whole time for this stuff to provide the best quality possible
oskarth
2018-04-16 11:19
Agree. Is this captured somewhere?
oskarth
2018-04-16 11:19
E.g. in issues with label or google doc or so
adamb
2018-04-16 11:19
We have some issues but it would be better to write all these stuff and set priorities
adamb
2018-04-16 11:19
and align it with the OKRs
oskarth
2018-04-16 11:22
good point
arnetheduck.slack
2018-04-16 11:32
nothing happened - still needs love!
arnetheduck.slack
2018-04-16 11:55
our own instance of http://hivetests.ethstats.net/ would be cool also
oskarth
2018-04-16 11:56
oh I like this
martin487
2018-04-19 18:15
Are we running geth nodes that we could use via RPC?
2018-04-19 18:15
@martin487 commented on @martin487?s file https://status-im.slack.com/files/U92A3R9A7/FA96J9Z7S/screen_shot_2018-04-19_at_20.12.17.png: Just learned that our TX dropping issues could be related to load balancing done by Infura.
igor680
2018-04-19 18:59
That makes sense, but is there a reason not to setup out own node? How many requests to them are there?
igor680
2018-04-19 19:01
We have status-cluster, but IDK if is suitable. Ping @adamb
igor680
2018-04-19 19:02
@martin487 can we just run a node on your server, like in a container? Then it's RPC/requests to it won't be exposed to anywhere.
martin487
2018-04-19 19:03
The issue with running a node on our server was that we didn't properly manage it, eventually it caused downtime
martin487
2018-04-19 19:04
While we might need to reconsider that it would be ideal if we would not need to worry about this and could consume it as a service so to say
igor680
2018-04-19 19:16
Yeah, but wouldn't you hit the same nonce issue if using any HA SAAS?
igor680
2018-04-19 19:22
Though, I don't have enough history. Did it work well before with Infura?
martin487
2018-04-19 19:24
interestingly it worked better when we were not properly incrementing nonces I think
igor680
2018-04-19 19:24
(moving it to a thread)
martin487
2018-04-19 19:25
then we pushed some changes to ?fix? this and suddenly most transactions go dropped
igor680
2018-04-19 19:25
but what was the problem you were solving by local nonce computation?
martin487
2018-04-19 19:26
since we're deploying identical contracts not increasing the nonces resulted in identical transactions when there should have been multiple
martin487
2018-04-19 19:26
(let me know if that makes sense)
igor680
2018-04-19 19:27
hmm, so you were trying to deploy multilple contracts nearly at the same time from the same account, right?
adamb
2018-04-19 19:29
How exactly are you doing that?
adamb
2018-04-19 19:29
pseudo code would be good
igor680
2018-04-19 19:31
I mean, theoretically, if the contract would be a bit different, it might help, right?
aka, if there is a "nonce" as a constant in a contract that is changed for every request.
then the data send will be different, and you won't need to increment nonces.
I don't know how would it work in your case, but that is how I would tried to "hack" this system :slightly_smiling_face:
martin487
2018-04-19 19:32
Let?s talk more about this tomorrow, it's late for all of us and we should not be working :slightly_smiling_face:
igor680
2018-04-19 19:33
maybe we can have a call about that tomorrow? @martin487 @adamb wdyt?
adamb
2018-04-19 19:33
Yeah, let?s continue tomorrow! We should isolate a problem and figure it out
adamb
2018-04-19 19:33
I can jump on a call
adamb
2018-04-19 19:33
Tomorrow :D
martin487
2018-04-19 19:51
Would greatly appreciate that - but I have a bunch of other things to attend tomorrow, would need to be past 4pm Berlin time. Let me know if that works
igor680
2018-04-19 19:54
4 pm works for me
dmitry.shulyak
2018-04-20 04:10
> I mean, theoretically, if the contract would be a bit different, it might help, right?
dmitry.shulyak
2018-04-20 04:10
it wont
dmitry.shulyak
2018-04-20 04:11
new contract address is derived from sender and nonce
dmitry.shulyak
2018-04-20 04:13
you will get either nonce too low or transation will replaced if same pair of sender + nonce will be used
dmitry.shulyak
2018-04-20 04:24
> interestingly it worked better when we were not properly incrementing nonces I think
dmitry.shulyak
2018-04-20 04:25
maybe you had retries when tx with known hash was returned? txs with same nonce definitely are dropped and cant "work"
igor680
2018-04-20 07:02
so it is essentially the address and the nonce that are in play there.
so if we take load balancing into account (we don't know to which node we land), idk how to solve this problem in a good way
martin487
2018-04-20 07:46
cc @vitaliy
martin487
2018-04-20 07:47
Maybe a reasonable quick fix could be to wait between sending transactions, or more involved, ensure the previous tax/nonce has been mined
igor680
2018-04-20 07:48
yeah, sounds like that will help if you don't have tons of contracts to deploy at a very short time
igor680
2018-04-20 07:48
just don't deploy anything until you have a receipt from the previous tx
adamb
2018-04-20 07:52
It does not need to be mined. Do you use `eth_getTransactionCount` method?
adamb
2018-04-20 07:52
to calculate `nonce`?
igor680
2018-04-20 07:54
@adamb but the problem was that they didn't know on what node they land because of load balancing. Is `eth_getTransactionCount(..., "pending")` local?
adamb
2018-04-20 07:57
No, it goes to the upstream node but load balancing may be an issue here. So, when the process starts for the first time, it must take the nonce using this method and then it can cache it and increment whenever a successful transaction is sent
adamb
2018-04-20 07:57
it can do `max(local, txCount)`
adamb
2018-04-20 07:57
the easiest would be to send transactions sequentially
dmitry.shulyak
2018-04-20 07:57
i think this is what they did after prev discussion, now there are more problems
adamb
2018-04-20 07:58
Can we see the code?
pedro
2018-04-20 08:28
Interested in this discussion, I ran into the same issues with the kudos bot payouts (did a quick hack of waiting for a couple of minutes between every transaction, as I also ran into errors even after incrementing the nonce sequencially and I didn't want to waste more time on it)
martin487
2018-04-20 08:56
Not on computer right now. For those who want to see code now search for `NonceTracker` in the openbounty codebase. Will also share a link later.
When we call `getTransactionCount` we use `LATEST` block, not `PENDING` ? I?m not entirely sure of implications
martin487
2018-04-20 08:57
I think we do pretty much what @adamb described with the difference above
adamb
2018-04-20 09:19
`LATEST` means mined blocks `PENDING` means tx not necessarily mined but received
adamb
2018-04-20 09:19
`PENDING` should be used
vitaliy
2018-04-20 10:06
hi, just read the thread. Some thoughts, might repeat what was already said:
1. We used to have no gaps before, because we always calculated nonce based on `eth_getTransactionCount`. Then we noticed that we might execute quick requests in succession with the same nonce obtained via`eth_getTransactionCount`, and get same tx hashes. We should have been refused all subsequent txs with the same nonce, but given Infura?s load balancing sequential requests might well have been processed by different nodes.
2. Whenever we invoke `eth_getTransactionCount`, we are not sure if this the canonical one because of load balancing. The value returned might differ depending on where the request landed in Infuraland. Do you think it is correct to say that?
3. Also, there can be 2 kinds problems with txs: they either get refused outright and we get an error during the invocation of `eth_sendRawTransaction`, or they might not get mined after we get a tx hash. The former we can catch, but how do we reliably check for the latter? The problem is (as I see it) that we can know for sure that a tx has been mined if there?s something in `eth_getTransactionReceipt`, but how do we know if tx was dropped? Check for some time, like an hour or 30 mins? Or verify block counts? What do you think?
4. Now we changed the logic so we increment nonces ourselves, therefore introducing ?speculative? nonces. Because of 2 and 3 we never know for sure if the nonces were right and/or will get mined. Even without speculative nonces we had the same problem, but it manifested itself less
5. Good thing is that we have the logic in place to check for tx errors, there are scheduler threads that try to get tx receipts each minute. So we can use them to check for unmined txs and therefore release associated nonces, if a tx was dropped from the pool.
6. The objective would then be to decrease the amount of failed txs, this means essentially the amount of wrong nonces we submit. The problem is interesting, given the amount of uncertainty. I like it though)
7. There has to be item number 7. Most primitive nonce speculation logic would be for instance to rely on our replay features and use no speculative nonces, always relying on `eth_getTransactionCount`
vitaliy
2018-04-20 10:07
please correct me if any of the above points is wrong
martin487
2018-04-20 11:46
1. :white_check_mark:
2. :white_check_mark:
3. +1 on verifying if transactions have been successful using `eth_getTransactionReceipt`. Ideally we should track all transactions we do and verify that they are executed properly, retry if not.
4. I wouldn't call them speculative. After all they are nonces for our account and we can fairly certainly say that they are correct. The issue, as you said, is that different geth nodes might have different views on that.
5. :+1:
6. Yeah. As outlined by others here, options seem to be: a) Queue for deploys and wait until previous TX hash a receipt or b) Communicate with a single geth node.
7. If we use option a) from point _6._ my understanding is that we could safely rely on `eth_getTransactionCount`
adamb
2018-04-20 11:53
What we do in status-go is that we keep the nonce locally, per each address, and sync with the upstream using `eth_getTransactionCount`. Whenever a new transaction is sent, we make a request for the upstream transaction count and do `localNonce = max(localNonce, upstreamNonce)`. If the request is send properly, we increment `localNonce`. I think we block for the time the request is sending so it's not possible to concurrently send two transaction from one address.
I would recommend to use `eth_getTransactionCount` with `pending`.
igor680
2018-04-20 11:55
I just noticed from my experiments with a private network, that `eth_getTransactionCount` on *non-mining* full nodes will return a correct nonce if you don't send transactions literally at the same time.
igor680
2018-04-20 11:55
if the node is mining, that won't work :slightly_smiling_face:
adamb
2018-04-20 11:56
with INFURA load balancing there might be a bit more delay but still I do not expect it to be higher than 1-2 seconds
igor680
2018-04-20 11:56
theoretically we can send a tx and wait until the nonce number increases :slightly_smiling_face:
adamb
2018-04-20 11:57
yeah, depends on a use case
growbot
2018-04-20 14:19
has joined #devops
pedro
2018-04-20 14:22
kudos @jacqueswww for always being responsive when we need anything in Scaleway/Digital Ocean! You're awesome!
martin487
2018-04-20 14:24
Given that we have a couple of strategies here I figure we probably don't need a call anymore?
vitaliy
2018-04-20 14:26
agree there?s no need for a call for now
jakub
2018-04-26 15:01
question: why doesn't `deploy-cluster` playbook not include the `common` role that sets up users and the like?
adamb
2018-04-26 15:19
No particular reason. It should
jakub
2018-04-26 15:19
kay, I'll add it in the PR that also adds ssh keys
oskarth
2018-04-30 01:56
Is it just me or has there not been a nightly build since Friday? https://status-im.slack.com/archives/C8RNGLKRT/p1524799655000095 see #jenkins
nastya
2018-04-30 06:23
please check http://artifacts.status.im:8081/artifactory/nightlies-local/ healthcheck
oskarth
2018-04-30 07:13
@jakub if you are working today (holiday in some places), would you mind having a look at this?
nastya
2018-04-30 07:17
@oskarth can i kill the build ?
jakub
2018-04-30 07:28
I'm here, but I am back home at like 4am and just woke up, give me maybe 40 min to get some breakfast, I'm starving
nastya
2018-04-30 07:29
sure! :slightly_smiling_face: :coffee::cookie:
oskarth
2018-04-30 08:03
Sure! X2
oskarth
2018-04-30 08:03
Seems like artifactory
jakub
2018-04-30 08:18
okay, that was some good blood sausage, now let me check it out
jakub
2018-04-30 08:20
okay, so nightly #358 seems to be waiting for 2 days to get macos host which has been running which has been running PR-3860 #11 for over 2 days
jakub
2018-04-30 08:21
looks stuck on `:app:compileReleaseSources`
jakub
2018-04-30 08:22
hmm #10 worked just fine
nastya
2018-04-30 08:23
http://artifacts.status.im:8081/artifactory/nightlies-local/ please healthcheck this
jakub
2018-04-30 08:23
good idea
jakub
2018-04-30 08:24
oh look, it
jakub
2018-04-30 08:25
it's yet another cloud provider we use
oskarth
2018-04-30 08:26
I think Artifactory is still hosted on the SOB box
oskarth
2018-04-30 08:26
:see_no_evil:
jakub
2018-04-30 08:26
mmmm, nothing like a nice service cocktail
nastya
2018-04-30 08:26
:thisisfine:
jakub
2018-04-30 08:28
Looks like @jarradhope knows how to get to Linode console, without that I can't do much
jacqueswww
2018-04-30 08:30
what happens if we just kill https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-3860/11/ for now?
jakub
2018-04-30 08:31
I guess there's no harm, but it won't fix the build
jakub
2018-04-30 08:31
so I guess it's gonna get stuck again
jacqueswww
2018-04-30 08:31
ah ok
andrey
2018-04-30 08:32
@adamb @divan @roman i remember you fixed http://artifactos.status.im maybe you have access ?
divan
2018-04-30 08:33
Nope, sorry, I don't have access there :confused:
oskarth
2018-04-30 08:35
Did we move it from SOB machine?
jakub
2018-04-30 08:36
```
% dig +noall +answer http://openbounty.status.im http://artifacts.status.im
http://openbounty.status.im. 257 IN A 139.162.11.12
http://artifacts.status.im. 270 IN A 139.162.11.12
```
jakub
2018-04-30 08:36
nope
jakub
2018-04-30 08:36
same IP
oskarth
2018-04-30 08:42
do you have access to it yet?
oskarth
2018-04-30 08:42
Cc @martin487 @vitaliy
jakub
2018-04-30 08:42
not really, Jacques doesn't seem to have it, he told me to poke Jarrad but he seems out
oskarth
2018-04-30 08:43
No need for Jarrad
oskarth
2018-04-30 08:43
If SOB ppl not online I can give soon ish
vitaliy
2018-04-30 08:43
maybe related to the host being down on Friday (https://status-im.slack.com/archives/C5Z5NRYKS/p1524852943000679)
vitaliy
2018-04-30 08:44
OpenBounty is up since Friday, it was restarted automatically after the host downtime
vitaliy
2018-04-30 08:44
does anyone know what happened there?
vitaliy
2018-04-30 08:44
jfrog is down?
jakub
2018-04-30 08:45
what is jfrog?
vitaliy
2018-04-30 08:48
artifactory
jakub
2018-04-30 08:48
oh
adamb
2018-04-30 08:57
I can upload ssh keys etc.
adamb
2018-04-30 08:57
btw. why artifactory server is named `commiteth`?
jakub
2018-04-30 08:58
no idea, but if you can help me access it that would be great
andrey
2018-04-30 08:59
commiteth is a prev name for openbounty
vitaliy
2018-04-30 08:59
@adamb most probably because it was originally an open-bounty server (commiteth being its old name)
oskarth
2018-04-30 09:00
last time the problem was out of disk space
adamb
2018-04-30 09:00
@jakub done
oskarth
2018-04-30 09:00
even got to the point where we couldn't use standard unix tools, ended up nuking a bunch of folders and had to rebuild status-go and status-react artifacts
adamb
2018-04-30 09:01
it's fine, 52% used
oskarth
2018-04-30 09:01
not sure if this is the problem this tiome around, there's also some kind of clean up job configured in UI in artifactory IIRC
oskarth
2018-04-30 09:01
ah ok, different problem then
jakub
2018-04-30 09:02
@adamb thanks man, I've logged in, let me check it out now
jakub
2018-04-30 09:10
Looks like it failed around 6pm on the 27th
jakub
2018-04-30 09:10
```
2018-04-27 17:45:21,235 [ACCEPTED DOWNLOAD] nightlies-local:im.status.ethereum-ef57a5.apk for anonymous/207.46.13.202.
```
jakub
2018-04-30 09:10
last access log
jakub
2018-04-30 09:15
```
"State": {
"Status": "exited",
"Running": false,
"Paused": false,
"Restarting": false,
"OOMKilled": false,
"Dead": false,
"Pid": 0,
"ExitCode": 137,
"Error": "",
"StartedAt": "2018-03-12T19:14:30.206355349Z",
"FinishedAt": "2018-04-27T18:03:13.181865752Z"
},
```
jakub
2018-04-30 09:15
`OOMKilled` is false so not memory limits, I'll have to check what `137` exit cod emeans for artifactory
jakub
2018-04-30 09:17
I guess a `SIGKILL`? weird
jakub
2018-04-30 09:18
logs have no errors, I'll just restart it for now
jakub
2018-04-30 09:21
grrr, bash history has no timestamps...
jakub
2018-04-30 09:23
http://artifacts.status.im:8081/artifactory/nightlies-local/ - looks like it's back up
jakub
2018-04-30 09:25
and it looks like https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-3860/11/ finished
jakub
2018-04-30 09:26
which means https://jenkins.status.im/job/status-react/job/nightly/358/ finally got a slave so it's running now
jakub
2018-04-30 09:26
jesus, the build queue is insane
jakub
2018-04-30 09:26
I'll kill some builds to not waste time with old commits
oskarth
2018-04-30 09:26
feel free to kill all old ones
jakub
2018-04-30 09:26
yeah
jakub
2018-04-30 09:28
also, we should probably add some more slaves to jenkins
jakub
2018-04-30 09:28
servers are cheap, engineers time isn't
oskarth
2018-04-30 09:28
100%
jakub
2018-04-30 09:29
I'll get on that today, but it's gonna be only linux hosts since macos is a different beast
jakub
2018-04-30 09:29
I'm not even sure where that one macos machine is hosted
jakub
2018-04-30 09:34
yet another cloud provider
jakub
2018-04-30 09:44
Looks like nothing was done to the container at the time of it's failure:
```
root@commiteth:/var/log# docker events -f container=af47aac4c279 --since '2018-04-24'
2018-04-30T17:19:05.495284354+08:00 container start af47aac4c2798b3965ca2692db95e10f3638fe82c41a6755f1da1c207ba38a2c (image=http://docker.bintray.io/jfrog/artifactory-oss:latest, name=artifactory)
2018-04-30T17:19:05.495317785+08:00 container restart af47aac4c2798b3965ca2692db95e10f3638fe82c41a6755f1da1c207ba38a2c (image=http://docker.bintray.io/jfrog/artifactory-oss:latest, name=artifactory)
```
So I have no idea why it stopped, has this happened before?
oskarth
2018-04-30 09:45
not sure actually, but possible
oskarth
2018-04-30 09:45
maybe some basic service monitoring + (optional) auto restart would help for future?
oskarth
2018-04-30 09:45
assuming it is a bit janky
jakub
2018-04-30 09:46
sure, you get that by just using `--restart` with `docker run`, I see there's a `run.sh` script in the `commiteth` home dir that starts the container
jakub
2018-04-30 09:47
I can modify it but it's just a temporary fix, this thing needs a proper installation and be run as service
oskarth
2018-04-30 09:47
yeah
oskarth
2018-04-30 09:47
and probably put on a difference box^Hplace
jakub
2018-04-30 09:48
true dat
nastya
2018-04-30 10:12
@jakub any harm if i manually kill builds? queue is crazy we can wait forever
jakub
2018-04-30 10:12
please do
jakub
2018-04-30 10:12
I wasn't sure which ones I could kill so just killed ones running old builds
jakub
2018-04-30 10:12
I'll be adding more hosts when I figure out how to get more macos hosts
dmitry.shulyak
2018-05-03 16:24
@adamb @pedro we need to update mailservers with newer statusd, before updating status-react with newer status-go next time
dmitry.shulyak
2018-05-03 16:25
because of bloom filters
dmitry.shulyak
2018-05-03 16:26
i don't have access to those machines, but we can use that moment to switch every network to one based on discovery
pedro
2018-05-04 13:39
@martin487 after a discussion in https://status-im.slack.com/archives/C8QP8S5UH/p1525428110000434, I think we really need to start getting the ball rolling on moving to a single CI system. I've started a document at https://docs.google.com/document/d/1fh97IaWVzC1Z3jq--xsC7mmWZ_mnjzGilk9J6QQ9His/edit?usp=sharing to gather requirements for a new unified platform, it would be great if you could contribute with your team's needs in order to get a complete picture.
dmitry.shulyak
2018-05-04 13:43
@pedro btw is teamcity is self hosted or on-prem?
pedro
2018-05-04 13:43
there's both possibilities
pedro
2018-05-04 13:44
I've worked on OSS projects with the self hosted, and on day jobs with the on-prem
yenda
2018-05-06 21:37
@jakub my PR was merged but is the change deployed ? https://github.com/status-im/status-cluster/pull/82/commits
```
debug1: Next authentication method: password
eric@51.15.35.110's password:
```
jakub
2018-05-07 05:08
I believe @adamb did deploy it, but let me make sure
jakub
2018-05-07 05:18
I ran `ansible-playbook users.yml` but it seem like I still lack access to most hosts so we'll have to wait for @adamb to do it
jakub
2018-05-07 05:19
also, there's so many places where use of `root` as `remote_user` in our ansible, yet another thing to fix
jakub
2018-05-07 05:29
yep, `ansible all -e ansible_ssh_user=jakub -m ping -o` fails for me for most hosts
jakub
2018-05-07 05:29
I'll be back around 11:00 am CEST
adamb
2018-05-07 07:30
@yenda with `eric@51.15.35.110` you should have access now. I didn't run it for this host, only for Ropsten/Rinkeby
adamb
2018-05-07 07:32
@jakub there is a problem with `uid`s and the way this role works though. This host had other users created in the meanwhile and `6005` and `6006` are taken which are assigned to `vitaliy` and `jakub` users in `users-dev.yml`
jakub
2018-05-07 08:17
When I get back I'll update the users role to maybe bump all users to 7k and start counting again. Also I'm reseraching use of etcd for ansible dynamic inventory which will also manage users.
adamb
2018-05-07 08:50
I am not sure if it will help. Unless we make sure that e.g. service users use 6XXX and regular users 7XXX spaces
adamb
2018-05-07 08:50
then it should be fine
jakub
2018-05-07 08:52
Thats the idea
roman
2018-05-08 05:13
> stderr: error: could not lock config file .git/config: No space left on device
roman
2018-05-08 05:13
^ @jakub
jakub
2018-05-08 06:10
I'm out today dude, sorry
arnetheduck.slack
2018-05-08 09:08
for swarm conference, mainframe has set up a "decentralized" chat on top of swarm PSS - http://sos18.mainframe.com/ - however, it requries some AWS deployment - do we have any aws expert around that would be kind enough to do that part of it?
igor680
2018-05-08 09:10
I was working with AWS :slightly_smiling_face: What exactly do we need to do?
igor680
2018-05-08 09:11
ah, I see, they have a CloudFormation stack script there
arnetheduck.slack
2018-05-08 09:12
yeah, and at the end of that process, it supposedly gives some magic string I need to pop into their app :slightly_smiling_face:
igor680
2018-05-08 09:13
but do we have an AWS account we want to deploy it to?
igor680
2018-05-08 09:21
because I'm not sure we have any AWS account for Status (cc @jacqueswww)
jacqueswww
2018-05-08 09:22
Not to my knowledge no
themue
2018-05-08 09:30
Will take a look at it too. Worked with AWS at Canonical.
igor680
2018-05-08 09:31
I mean, essentially we need an account to deploy this stuff to, and that's it :slightly_smiling_face:
igor680
2018-05-08 09:32
Otherwise, it is just a deployment from a template.
themue
2018-05-08 09:32
Still got my private one but don't know about a company one.
arnetheduck.slack
2018-05-08 09:34
no idea about anything AWS - apparently the template deploys a free thing - this is nothing permanent - just interesting as a test for the next 3 days
igor680
2018-05-08 09:34
@themue can you deploy it to your private one then? it's just `t2.micro`, should be within a free tier?
themue
2018-05-08 09:35
Will do.
igor680
2018-05-08 09:35
+ 1 bucket with a certificate, if I'm reading the script right :slightly_smiling_face:
igor680
2018-05-08 09:36
thanks! :slightly_smiling_face:
themue
2018-05-08 10:15
So, Stack is up and running and I'm in. Web Socket URL is
arnetheduck.slack
2018-05-08 10:16
fantastic, thanks! just tried the app and it's asking for security certificates as well.. 3 `.pem` files, do you see something like that?
themue
2018-05-08 10:17
Yes, just looking where to place them for you.
arnetheduck.slack
2018-05-08 10:17
slack here? security by obscurity is probably fine in this case :slightly_smiling_face:
themue
2018-05-08 10:19
For Onyx testing.
themue
2018-05-08 10:19
For Onyx testing.
themue
2018-05-08 10:19
For Onyx testing.
themue
2018-05-08 10:37
Somebody in?
adamb
2018-05-08 11:59
What is this?
themue
2018-05-08 12:12
@adamb See @arnetheduck.slack and the thread below. I quickly created a node for Onyx on my private AWS for testing. It's distributed chat based on PSS.
igor680
2018-05-08 12:12
"distributed", heh :laughing:
themue
2018-05-08 12:13
Yeah, only the nodes, not the clients.
adamb
2018-05-08 12:15
How is it distributed exactly?
adamb
2018-05-08 12:15
the fact that I can install it on my account makes it distributed? :smile:
themue
2018-05-08 12:15
Not yet really sure if the nodes are distributed and can run multiple clients or if I always need one node for per client.
themue
2018-05-08 12:16
Not comparable to the ideas of Whisper.
adamb
2018-05-08 12:17
PSS is comparable but they way it's used here does not look decentralized. Unless it's possible to somehow dynamically create new nodes and discover them by the client.
themue
2018-05-08 12:20
Yip, didn't meant PSS but how it is used here so far. Version is 0.4.2 and I'm looking for their plans.
adamb
2018-05-08 12:21
But the overall experience looks nice. I would love to have a similar easy way to set up Status cluster with bootnodes, Whisper nodes, MailServer and so on
themue
2018-05-08 12:24
Would be an interesting task, yes.
dmitryn
2018-05-08 12:56
@adamb https://github.com/status-im/status-cluster/pull/85 could you review this and run ansible scripts to deploy my ssh key please?
adamb
2018-05-08 13:00
Where would you like to deploy it?
adamb
2018-05-08 13:00
Also there is a bug in the PR
dmitryn
2018-05-08 13:02
@adamb deploy to mailserver, i?d like to check its logs. Bug with `uid` is fixed btw
arnetheduck.slack
2018-05-08 15:11
works! cheers @themue and @igor680!
dmitryn
2018-05-08 18:53
@adamb could you suggest where to find mailserver logs, is it somewhere in 51.15.35.110:/var/mailserver-8ca5b242/ ?
adamb
2018-05-08 18:54
@dmitryn you need to use docker logs command
adamb
2018-05-08 18:55
It?s only logged to stdout
jakub
2018-05-11 10:22
hey everyone, I've created a repo for managing infra/devops related passwords using password-store: https://github.com/status-im/infra-pass
jakub
2018-05-11 10:22
access is on case-by-case basis and requires some GPG magic to make work so poke me if you need it
jeluard
2018-05-11 11:48
Can't access it
jeluard
2018-05-11 11:48
Ah I need black magic :slightly_smiling_face:
jakub
2018-05-11 11:48
yeah, I'll have to add you to people who can see it
jakub
2018-05-11 11:48
give me a min
jakub
2018-05-11 11:51
You should see the repo now, if you need access then you can cut an issue and specify why you need it, and does it have to be read-only or read/write
jeluard
2018-05-11 11:52
Cool thanks!
jakub
2018-05-14 08:14
ey yo! I've made another thing, here's https://github.com/status-im/infra-hq which currently only deploys an configures an etcd cluster for a dynamic inventory
jakub
2018-05-14 08:15
later we can add to it things like monitoring dashboard, logging dashboard, alarming and other meta systems like that
jakub
2018-05-14 08:15
It depends on infra-pass for getting necessary secrets, I'm in the process of writing a README for infra-hq right now
jakub
2018-05-14 13:53
I've moved the `common` role(which grants access for users) from `status-cluster` to a separate repo: https://github.com/status-im/infra-role-common , this way we can re-use it in different projects like `infra-hq` or `infra-eth-cluster`
jakub
2018-05-14 14:00
Or rather than moved I've copied and adjusted, slowly we'll move away from using `status-cluster` completely
jakub
2018-05-14 14:16
question: I'm starting work on `infra-eth-cluster` and I'm trying to figure out what is the correct starting point, is it https://github.com/status-im/status-cluster/blob/master/deploy-cluster.yml ?
jakub
2018-05-14 14:18
I know there's also https://github.com/status-im/status-cluster/blob/master/discovery.yml but I'm not sure how they are related
jakub
2018-05-14 14:18
as far as I can tell `deploy-cluster.yml` deploys the mainnet cluster
jakub
2018-05-14 14:19
and `discovery.yml` is the new setup that uses boot nodes, is that right?
adamb
2018-05-14 14:46
Yeah, it's correct.
adamb
2018-05-14 14:46
I would start with bootnodes
adamb
2018-05-14 14:46
provision them and make sure they are running
adamb
2018-05-14 14:46
next we can do Whisper nodes
adamb
2018-05-14 14:47
and finally MailServers
jakub
2018-05-14 14:47
yeah, about that, I'm trying to figure out what's the proper setup for such nodes and I'm not sure who to talk to
jakub
2018-05-14 14:48
I've noticed that some of our nodes run multiple docker containers
adamb
2018-05-14 14:48
Yes, that's true
jakub
2018-05-14 14:48
some more, some less, some with mail servers some without
adamb
2018-05-14 14:48
that was kind of experimenting
jakub
2018-05-14 14:48
and I want to get a more clear picture of what a "perfect setup" looks like
adamb
2018-05-14 14:48
with bootnodes, we probably want one bootnode per server
jakub
2018-05-14 14:48
when someone writes "terraform apply" what do they want to see after it finishes
adamb
2018-05-14 14:49
I see. So yeah, with bootnodes, we want one per server as they are very lightweight
jakub
2018-05-14 14:49
is there some kind of golden ration of mail nodes to whisper nodes? or do we not know that?
adamb
2018-05-14 14:49
and the only thing is to provide HA so distribute them between locations
adamb
2018-05-14 14:49
MailServer is a single entity
adamb
2018-05-14 14:49
We may end up having 2-3 of them but it's not scalable
adamb
2018-05-14 14:50
We scale only with Whisper nodes for now
jakub
2018-05-14 14:50
so `discovery.yml` is the current way we deploy a cluster, and `deploy-cluster.yml` is the old way?
adamb
2018-05-14 14:51
yeah, `deploy-cluster.yml` does not support discovery
jakub
2018-05-14 14:52
right, but we still want the monitoring parts that it deploys on the new nodes
jakub
2018-05-14 14:53
also, I see there's https://github.com/status-im/status-cluster/blob/master/mailserverw6.yml
jakub
2018-05-14 14:53
what's the connection between `mailserverw6` and `whispermail` ?
adamb
2018-05-14 14:53
Right, on the servers running bootnodes we also want to run monitoring processes
adamb
2018-05-14 14:54
mailserverw6 is older and kind of Whisper V6 experiment
adamb
2018-05-14 14:54
so I would look at `whispermail`
jakub
2018-05-14 14:54
I see, so the `whispermail` role is the current iteration of that
adamb
2018-05-14 14:55
yea
jakub
2018-05-14 14:56
I find the term mailserver to be confusing, since the `whispermail` role does not configure a mail server, it configures `statusd` with mail-related options
adamb
2018-05-14 14:58
yeah, exactly, mail server is `statusd` process with some mail-related options
jakub
2018-05-14 14:59
okay, so for now we want a single statsd service per host or do we want to have multiple? Should the hosts running whispermail also have regular statsd containers or not?
adamb
2018-05-14 15:02
`statusd`* :smile: we need to have two options. Let?s run a single bootnode per single host. And N Whisper nodes per single host.
adamb
2018-05-14 15:02
Let?s not care about mailservers for now
jakub
2018-05-14 15:04
right, I'm trying to figure out what kind of variables for terraform we want, I assume we'd want `hosts_count` and `mail_hosts_count`, and in addition to that `whisper_nodes_per_host`
adamb
2018-05-14 15:05
makes sense
jakub
2018-05-14 15:07
cool, thanks for the clarifications, this helps a lot
adamb
2018-05-14 15:08
no problem!
anna
2018-05-16 10:35
there is big queue of PRs now on Jenkins- any known reason?
anna
2018-05-16 10:36
^ @jakub
jakub
2018-05-16 10:37
yeah, theres a pr request that keeps building for the last few hours
jakub
2018-05-16 10:37
I'll kill it for now
jakub
2018-05-16 10:38
It was stuck on iOS build step, weird, no errors
anna
2018-05-16 11:18
@jakub same issue for the next PR builds? Build iOS takes 23 min already on https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-4289/
anna
2018-05-16 11:22
I?m not sure but can be https://github.com/status-im/status-react/pull/4169 the reason that there is some issue with Jenkins build step on iOS @jeluard?
jeluard
2018-05-16 11:23
No this is an alternate build mechanism not used yet
anna
2018-05-16 11:23
ok, sorry for ping then
jeluard
2018-05-16 11:26
:+1:
igor680
2018-05-16 12:09
btw, can you ssh to the macos runners and make sure that there are no stuck processes? I see that when the build is cancelled...
`Could not connect to macos1 to send interrupt signal to process`
igor680
2018-05-16 12:09
maybe just the runner is in a bad shape...
jakub
2018-05-16 12:11
Looks correct to me, a bunch of `clang` and `xcodebuild` processes running
jakub
2018-05-16 12:11
which matches what the current build runs are doing
jakub
2018-05-16 12:11
but let me check the process tree
jakub
2018-05-16 12:12
yeah, it looks correct, all clang processes come from the 2 builds running right now
igor680
2018-05-16 12:13
what about RAM/disk space/CPU usage? are they good?
jakub
2018-05-16 12:14
`Load Avg: 11.44, 10.16, 6.72`
jakub
2018-05-16 12:14
sounds correct considering we have 12 cores
jakub
2018-05-16 12:14
memory is half used, half cache
igor680
2018-05-16 12:16
ok, if on the release branch is ends in the promised 7 minutes, then it isn't the runner :slightly_smiling_face:
igor680
2018-05-16 12:18
otherwise we might probably want to try rebooting it, I think
jakub
2018-05-16 12:18
worth a try
jakub
2018-05-16 12:23
Weird, both compilations seem to be running but the cpu usage is pretty low now
jakub
2018-05-16 12:24
I can see the progress in the logs, but why is the resource usage so low
jakub
2018-05-16 12:25
I'm gonna reboot it, this looks really weird
igor680
2018-05-16 12:29
yeah, also please check the disk space, just to be sure :slightly_smiling_face:
jakub
2018-05-16 12:30
oooooooooooh, it's the freakin' inodes
jakub
2018-05-16 12:30
I think, this is kinda weird, macs aren't my thing
igor680
2018-05-16 12:31
ok, so is the fs super-slow?
jakub
2018-05-16 12:31
doesn't seem to be
jakub
2018-05-16 12:32
but let me check
igor680
2018-05-16 12:32
:disappointed: ok, I just didn't understand your comment about inodes
jakub
2018-05-16 12:32
```
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /home
```
jakub
2018-05-16 12:32
I saw this
jakub
2018-05-16 12:32
```
Filesystem Size Used Avail Capacity iused ifree %iused Mounted on
/dev/disk0s2 233Gi 110Gi 123Gi 48% 3872669 4291094610 0% /
devfs 179Ki 179Ki 0Bi 100% 618 0 100% /dev
map -hosts 0Bi 0Bi 0Bi 100% 0 0 100% /net
map auto_home 0Bi 0Bi 0Bi 100% 0 0 100% /home
```
jakub
2018-05-16 12:33
but apparently on macs dirs like home are not regular filesystems
jakub
2018-05-16 12:33
but virtual
jakub
2018-05-16 12:33
so inodes don't mean anything
igor680
2018-05-16 12:34
ah, that is fine, 123G available is fine
igor680
2018-05-16 12:34
so, let's restart it and see what happens
jakub
2018-05-16 12:34
already did
jakub
2018-05-16 12:34
9 mins ago
igor680
2018-05-16 12:34
ok, we will try to run a build on it
jakub
2018-05-16 12:35
I restarted this
jakub
2018-05-16 12:35
I assume release_upload should also run, or is that for later?
igor680
2018-05-16 12:35
It should, but we better restart it manually
jakub
2018-05-16 12:35
I just did Replay
jakub
2018-05-16 12:35
so it should be exactly the same as previous
igor680
2018-05-16 12:36
no-no, we want to run it manually w/o iOS build
igor680
2018-05-16 12:36
we only need to upload android
jakub
2018-05-16 12:36
aaah, I see
jakub
2018-05-16 12:36
okay, then feel free to kill it
igor680
2018-05-16 12:37
I see it is already aborted
igor680
2018-05-16 12:45
@jakub ok, we are running a release job now, so let's not touch this runner until has published the GP build.
jakub
2018-05-16 12:45
yeah
igor680
2018-05-16 12:45
thanks!
igor680
2018-05-16 12:53
@jakub the restart helped, thanks!
igor680
2018-05-16 12:53
andrey
2018-05-16 13:30
now Adnroid builds stoped with error
igor680
2018-05-16 13:31
I guess they were stopped manually to make some way to our struggling release upload
andrey
2018-05-16 13:31
ok
oskarth
2018-05-16 13:56
Thread about desktop CI ?> @vitaliy @volodymyr.kozieiev @jakub
oskarth
2018-05-16 13:58
Desktop would ideally like to have a job running on macos. WDYT about using current VNC setup to enable this pipeline? It?s a bit broken but it?d be useful to them. At the same time we don't want to spend too much time on this while we have critical beta blockers.
jakub
2018-05-16 14:01
Consul cluster just started working, you can check out the UI here: http://188.166.32.215/ui/
It's gonna replace `etcd` I configured before as inventory, it's better suited for multiple providers/data centers and covers healthchecks too
jakub
2018-05-16 14:02
I should probably put some github based auth in front of it
maxr
2018-05-16 14:03
@oskarth since we're working on build automation at the moment, I can start looking to configure dedicated MacOS machine with Jenkins Agent on it. And later, what if to add Ubuntu and Windows agent machines to do builds?
oskarth
2018-05-16 14:04
Sounds reasonable, let?s just make sure this syncs up with @jakub plans
jakub
2018-05-16 14:05
another macos machine sounds like a good idea
oskarth
2018-05-16 14:07
Where do we want to host it? Same place so we can use it for mobile builds too?
jakub
2018-05-16 14:07
I believe we already have a provider, MacStadium
oskarth
2018-05-16 14:07
Ideally the desktop and mobile pipelines are in sync cc @vitaliy
oskarth
2018-05-16 14:08
Since then product teams and QA can check behavior on one PR for all three platforms
maxr
2018-05-16 14:43
If I would like to start with it, what I need for this? 1. My own credentials to Jenkins with sufficient rights to create new Pipeline 2. Login credentials to new MacOS machine?
igor680
2018-05-16 17:38
Nightlies are getting stuck on trying to upload to Artifactory. can anyone take a look?
jakub
2018-05-16 19:48
Artifactory container looks fine, but I restarted for the placebo effect
jakub
2018-05-16 19:48
but yeah, it took almost an hour to get theough the deploy android stage
roman
2018-05-17 03:31
nightly is broken https://status-im.slack.com/archives/C8RNGLKRT/p1526525948000175
anna
2018-05-17 06:51
^ @jakub can you have a look why last nightly build failed, please?
anna
2018-05-17 06:51
```[nightly] Running shell script
+ git tag -d 0.9.16-189-g436ac1d7-1-gc80d4bd6
error: tag '0.9.16-189-g436ac1d7-1-gc80d4bd6' not found.
[Pipeline] }
[Pipeline] // stage
[Pipeline] slackSend
run slackstepsend, step null:false, desc BZU9o3LpIcV1ST0RFDhXjh7L:true
Slack Send Pipeline step configured values from global config - baseUrl: true, teamDomain: true, token: true, channel: true, color: false
[Pipeline] }
[Pipeline] // node
[Pipeline] End of Pipeline
ERROR: script returned exit code 1
Finished: FAILURE```
jakub
2018-05-17 07:25
Let me take a look
jakub
2018-05-17 07:25
Oh yeah, that, we talked about it yesterday, let me finish my tea and I'll make a PR
jakub
2018-05-17 07:31
the more people check this out the better, but as far as I can tell these are pointless
igor680
2018-05-17 07:54
there is already a PR for that
igor680
2018-05-17 07:55
they are evil! :evil:
lukasz949
2018-05-17 09:13
@roman can you review it ^
igor680
2018-05-17 09:20
@jakub @adamb re: ci and macos1 runner. can we (since tomorrow) increase the number of parallel jobs on this macos runner from 2 to 4? as I understand that is a pretty powerful machine there.
jakub
2018-05-17 09:20
yeah, why not, if it slows down the builds we can lower it
jakub
2018-05-17 09:20
but I was planning to add another macos runner
igor680
2018-05-17 09:21
Then we might have 8 jobs at once! :slightly_smiling_face:
jakub
2018-05-17 09:21
since we will be developing a desktop app that will also be running builds and test on macos amongst other platforms
jakub
2018-05-17 09:21
Once I'm done with todays infra meeting I'll provision another macos machine
igor680
2018-05-17 09:21
I agree, but let's not do anything until tomorrow, we need to be able to submit a new release to TestFlight today
jakub
2018-05-17 09:21
although I am wondering what is better, 3 weaker machines or one powerful
igor680
2018-05-17 09:21
I usually go for one powerful and more jobs in parallel
jakub
2018-05-17 09:22
fine by me
jakub
2018-05-17 09:22
I can create the slave today but keep it from going into the poll until you do the release to TestFlight
roman
2018-05-17 09:23
revert what?
igor680
2018-05-17 09:23
review :slightly_smiling_face:
roman
2018-05-17 09:24
oh jeez, need to buy new lenses, thanks
igor680
2018-05-17 09:25
sounds good! let's do that
jakub
2018-05-17 11:06
Merged the fix.
lukasz949
2018-05-17 12:58
Hey, we still can't build nightly. There was
```
> git rev-parse refs/remotes/origin/develop^{commit} # timeout=10
> git rev-parse refs/remotes/origin/origin/develop^{commit} # timeout=10
Checking out Revision 41900da38c9b5be420774f4dac626bc9fc8ab4d6 (refs/remotes/origin/develop)
> git config core.sparsecheckout # timeout=10
> git checkout -f 41900da38c9b5be420774f4dac626bc9fc8ab4d6
Commit message: "Merge pull request #4305 from status-im/fix/nightly-build-tag-removal"
> git rev-list --no-walk c80d4bd62c0a3bbf0fa42ed5b4e3e766e3d79f6f # timeout=10
[Pipeline] sh
[nightly] Running shell script
Cannot contact macos1: java.lang.InterruptedException
+ git fetch --tags
```
https://jenkins.status.im/job/status-react/job/nightly/392/
I stopped it and started a new build
igor680
2018-05-17 13:06
Hmm, our PR jobs don't cancel each other? I see 3 jobs for the same PR in the build queue...
igor680
2018-05-17 13:12
and if we want to figure out what is wrong with the slow builds, this plugin is useful https://wiki.jenkins.io/display/JENKINS/Timestamper
igor680
2018-05-17 13:13
I'm going to increase runners count for our macos1 machine, because the waiting times are painful. Ping me if you disagree, I'll do it in 10 minutes.
igor680
2018-05-17 13:23
? done. 4 executors now
lukasz949
2018-05-17 13:38
https://jenkins.status.im/job/status-react/job/nightly/393/console is still waiting
igor680
2018-05-17 13:42
@jakub hmm, ok, our mac node has some troubles now, it seems. I'll revert executors to 2 again...
jakub
2018-05-17 13:43
Oh yeah, 26 min for android build
jakub
2018-05-17 13:43
so far
igor680
2018-05-17 13:43
@jakub can you try connecting to it? it looks like it just lost connectivity
jakub
2018-05-17 13:43
it's up
jakub
2018-05-17 13:43
but it doesn't do anything
igor680
2018-05-17 13:43
so we need to restart jobs, basically?
jakub
2018-05-17 13:43
I'll reboot it
igor680
2018-05-17 13:44
okay
nastya
2018-05-17 14:53
so Jenkins is still sick?
igor680
2018-05-17 14:55
@jakub you are upgrading it, right? :slightly_smiling_face:
jakub
2018-05-17 14:55
yeah
jakub
2018-05-17 14:55
the package download is insanely slow
jakub
2018-05-17 14:55
give it 10 min
nastya
2018-05-17 14:58
sure
jakub
2018-05-17 14:59
rebooting
jakub
2018-05-17 15:00
oh, neat, it didn't drop existing builds
nastya
2018-05-18 07:20
@jakub jenkins looks brand new!
nastya
2018-05-18 07:21
but iOS builds are very slow
jakub
2018-05-18 08:23
Even on the new hosts? Let me look
jakub
2018-05-18 08:27
we explicitly specify use of `macos1` instead of using generic label `macos`
jakub
2018-05-18 08:27
so we are not using new hosts =]
jakub
2018-05-18 08:29
Also, which builds are slow, I'm looking at https://jenkins.status.im/job/status-react/job/pull%20requests/job/PR-4239/ and it seems to be usual speed
nastya
2018-05-18 08:30
i was taking about this one
nastya
2018-05-18 08:30
now looks fine
nastya
2018-05-18 08:31
your presence fixed that! :slightly_smiling_face:
jakub
2018-05-18 08:31
it's the power of friendship!
jakub
2018-05-18 08:31
or something
nastya
2018-05-18 08:31
:all_the_things:
jakub
2018-05-18 08:31
well anyway, once we merge my fix people will have to cherry pick into their branches to take advantage of new hosts
pedro
2018-05-18 12:52
@jakub are you aware of this?
2018-05-18 12:53
@jakub commented on @pedro?s file https://status-im.slack.com/files/U8Q0C9ZH6/FAS5VCTGB/image.png: I don't see it
2018-05-18 12:53
@jakub commented on @pedro?s file https://status-im.slack.com/files/U8Q0C9ZH6/FAS5VCTGB/image.png: all slaves are online
pedro
2018-05-18 12:54
�\_(?)_/�
andrey
2018-05-18 12:55
do we know why builds randomly fail? one build failed on Andorid, after restart it failed on ios
igor680
2018-05-18 12:59
also, macos2 and macos3 are mostly idle when we have a pretty long queue
igor680
2018-05-18 13:08
@jakub would you mind if I disable swap monitoring to Jenkins nodes? it seems pointless on macOS
jakub
2018-05-18 13:08
sure
jakub
2018-05-18 13:08
go ahead
jakub
2018-05-18 13:10
There an issue with the new nodes that I can't seem to be able to figure out
jakub
2018-05-18 13:10
when the build runs `npm install` it gets stuck
jakub
2018-05-18 13:11
seemingly randomly, sometimes it gets far with downloading packages, sometimes it stops very early
igor680
2018-05-18 13:11
no local npm cache I guess
igor680
2018-05-18 13:11
you can try going to these nodes, checking copying our package.json there and running npm install
igor680
2018-05-18 13:12
manually :disappointed:
jakub
2018-05-18 13:15
hmmmm, worth a try, but I don't get how that's different from jenkins running it
igor680
2018-05-18 13:17
maybe you will have to do it multiple times too
pedro
2018-05-18 13:20
could it related to the problem with the npm version mentioned in https://wiki.status.im/Building_Status#Requirements ?
igor680
2018-05-18 13:21
ah yeah, maybe
igor680
2018-05-18 13:21
it should be a very specific node
igor680
2018-05-18 13:22
so probably we'll need nvm and the right version selected
pedro
2018-05-18 13:22
worth checking
jakub
2018-05-18 13:26
ooooh, that's interesting, thanks for that link, I'll try that
jakub
2018-05-18 13:26
(also, wtf is wrong with npm? this is basic functionality, seriously)
igor680
2018-05-18 13:27
try installing `nvm` and adding `nvm use 9.3.0` before running npm
jakub
2018-05-18 13:27
will do
jakub
2018-05-18 13:27
also, the steps to setup osx for jenkins use are a real pain in the ass
pedro
2018-05-18 13:28
fwiw, I'm on node 8.6.0 and npm 5.5.1
jakub
2018-05-18 13:28
manual installations of jdk, git and xcode is just sad, I just have to click thing, no way to easily automate it
jakub
2018-05-18 13:28
thanks Pedro!
igor680
2018-05-18 13:28
I use 9.3.0 and it works just fine :slightly_smiling_face:
igor680
2018-05-18 13:28
but the latest used to freeze
pedro
2018-05-18 13:29
Maybe it got fixed recently. I've had mixed results with 9.1 so I had to go back
pedro
2018-05-18 13:29
what npm version do you use?
igor680
2018-05-18 13:29
5.5.1
igor680
2018-05-18 13:29
that's the compatible one
pedro
2018-05-18 13:29
huh, thought you'd need to have a higher npm version for something more recent like 9.3.0
igor680
2018-05-18 13:29
`nvm 9.3.0` actually installs `npm` 5.5.1 and compatible node automatically
pedro
2018-05-18 13:30
let me try that
pedro
2018-05-18 13:31
the install went fine, just testing the build
jakub
2018-05-18 13:37
seems to be working so far
pedro
2018-05-18 13:43
yup, 9.3.0 works for me :thumbsup:
igor680
2018-05-18 15:19
@jakub
```
[react_pull_requests_PR-4337-GMOYSWBUTJOE37F6TRVHCXK73V6577UTNP5KYRE7QRGRZT2II7ZA] Running shell script
+ npm install
env: node: No such file or directory
script returned exit code 127
```
:disappointed:
jakub
2018-05-18 15:19
ah yes, the new nodes
jakub
2018-05-18 15:19
which build?
jakub
2018-05-18 15:22
yeah, this is a PATH issue
jakub
2018-05-18 15:23
I thought I fixed this, give me a moment
jakub
2018-05-18 15:52
this is insane, nothing I change affects jenkins `PATH`
jakub
2018-05-18 15:53
not `/etc/profile`, not `.bashrc`, not `.bash_profile`, not `env.groovy`, not node configuration in Jenkins
jakub
2018-05-18 15:53
kill me, just kill me now
jakub
2018-05-18 16:01
oh, it takes effect after a reboot
jakub
2018-05-18 16:01
right, `.bashrc` changes require a reboot... because that's logical...
jakub
2018-05-18 16:03
Okay, it should work now(well, `node` and `npm` should)
igor680
2018-05-18 16:23
Awesome! Thanks!
roman
2018-05-20 05:33
https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-4221/ we have kind of random errors here, three cases when ```
/Users/jenkins/workspace/react_pull_requests_PR-4221-IPVTYGVGTYOFRS62JKDTI536VDR6E2XGJUQ7KMQFZC4DRJRJDONA@tmp/durable-019d17cd/script.sh: line 2: pod: command not found`
```
and build #22 between them which passes this command. A bit weird
roman
2018-05-20 05:39
and now it passed again, weird
pedro
2018-05-20 07:32
Was it always in the same build agent/machine?
roman
2018-05-20 09:38
yes, build always happened on the same machine, idk if this was changed recently
adan
2018-05-20 12:08
I can't merge the PR https://github.com/status-im/status-react/pull/4348 due to `@MaxRis
continuous-integration/jenkins/branch ? This commit cannot be built` How can I fix the step?
jacqueswww
2018-05-20 16:09
LOL a reboot !?!?
jacqueswww
2018-05-20 16:09
just found this thread and I am chuckling my ass off.
jakub
2018-05-20 16:09
Yeah, later on I figured that the jenkins.jar process that is tarted by Jenkins master via SSH loads stuff from .bashrc and never reloads it
jakub
2018-05-20 16:09
and it doesn't respect SIGHUP
jakub
2018-05-20 16:09
so you have to kill it
jakub
2018-05-20 16:10
and restart it
jakub
2018-05-20 16:10
so not really a reboot
jacqueswww
2018-05-20 16:10
ah OK
jacqueswww
2018-05-20 16:10
but still, glad you managed though !
jacqueswww
2018-05-20 16:10
:slightly_smiling_face:
jakub
2018-05-20 16:11
my mistake was assuming wrongly(like many times before in my life) that a process would respect SIGHUP
jakub
2018-05-20 16:11
but I guess I demand too much =]
jacqueswww
2018-05-20 16:11
So our mac builds all running smoothly now?
jacqueswww
2018-05-20 16:11
=}
jakub
2018-05-20 16:11
they should be, but we got this working only on friday, so maybe something will come up again on Monday, so far so good
jacqueswww
2018-05-20 16:11
great stuff!
jakub
2018-05-20 16:11
but I still have to sit down and try to parallelize the build across multiple hosts
jakub
2018-05-20 16:11
I already have a branch where I'm working on that but it's not done
jakub
2018-05-20 16:12
I'll have to run the app build manually and see what artifacts it generates to correctly use `stash` and `unstash`
jacqueswww
2018-05-20 16:13
I am still sure this will still make a considerable improvement
jakub
2018-05-20 16:13
oh yeah
anna
2018-05-21 09:01
@jakub morning :slightly_smiling_face: for some reason most of PR builds are red. I've triggered new https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-4286/7/console and got ` * What went wrong:
Could not determine java version from ?10.0.1?. `
jakub
2018-05-21 09:04
Right, it's one of the new hosts, let me check
jakub
2018-05-21 09:07
Right, it was downloading an old version of gradle: `gradle-4.1-all.zip`
jakub
2018-05-21 09:08
I installed `4.7` on the new slaves, let's see now:
https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-4286/8/console
jakub
2018-05-21 09:14
Nope, fail again, hmmmm
jakub
2018-05-21 09:17
oh, it's Java version mismatch
igor680
2018-05-21 09:47
@jakub I took the "faulty" nodes offline. I think what we can do is:
- remove the 'macos' tag from them
- make a separate job that builds normal develop branch that runs on, say, macos2
- make sure that it passes
- add 'macos' tag back to 'macos2'
- repeat for 'macos3'
jakub
2018-05-21 09:47
I just downgraded JDK, it should be fine now
igor680
2018-05-21 09:48
let's try that on a separate job, otherwise it stalls the dev process
jakub
2018-05-21 09:48
okay
jakub
2018-05-21 09:48
what do you want me to test? nightly?
igor680
2018-05-21 09:49
yep, I changed the label to `macos_candidate` on these nodes
igor680
2018-05-21 09:49
so just build a nightly on this ~tag~ label
igor680
2018-05-21 09:56
actually, it might be easier with a PR build :slightly_smiling_face:
igor680
2018-05-21 09:58
I'll make a PR now :slightly_smiling_face:
jakub
2018-05-21 09:58
hmm, I'm trying to run the nightly but I'm getting git errors, weird
jakub
2018-05-21 09:58
the credentials are set, git is available, permissions are right...
igor680
2018-05-21 09:59
`Could not init /Users/jenkins/workspace/react_pull_requests_PR-4357-UEDTNANIQEEGI3UR3VPZ23GBNLNTFGI2IJLGELQNL5Q5IUPEGFUA`
igor680
2018-05-21 10:00
`Could not init /Users/jenkins/workspace/tests/nightly-test`
jakub
2018-05-21 10:00
yeah
igor680
2018-05-21 10:00
permissions?
jakub
2018-05-21 10:00
`Caused by: java.io.IOException: Cannot run program "git" (in directory "/Users/jenkins/workspace/tests/nightly-test"): error=2, No such file or directory`
igor680
2018-05-21 10:01
�\_(?)_/�
jakub
2018-05-21 10:01
yeah :smile:
igor680
2018-05-21 10:01
I need to run for an hour now, but you can also use this PR to check these runners: https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-4357/
jakub
2018-05-21 10:01
I'm trying `println "/usr/local/bin/git --version".execute().text` in host console
jakub
2018-05-21 10:01
and I get `java.io.IOException: error=0, posix_spawn failed`
jakub
2018-05-21 10:06
OKay, i think I got it
jakub
2018-05-21 10:06
after downgrading the ~JRE~ JDK I had to restart `jenkins.jar`
igor680
2018-05-21 10:07
Eh
igor680
2018-05-21 10:08
Did Jenkins just missed the Dockerization revolution? :)
jakub
2018-05-21 10:08
yeah, they did
maxr
2018-05-21 10:25
FYI `macos1` node is setup and used to build StatusIm Desktop with separate Pipeline ( creation of Pipeline is in progress )
adan
2018-05-21 10:39
@jakub I've faced new issue in e2e builds which is `ERROR: Error fetching remote repo 'origin'` https://jenkins.status.im/job/end-to-end-tests/job/status-app-end-to-end-tests/814/console
adan
2018-05-21 10:40
I did no recent changes, do you know what exactly caused the issue?
jakub
2018-05-21 10:40
hmmm: `stderr: error: cannot lock ref 'refs/remotes/origin/tests/maintenance': 'refs/remotes/origin/tests/maintenance/#2654' exists; cannot create 'refs/remotes/origin/tests/maintenance'`
adan
2018-05-21 10:41
should I delete refs/remotes/origin/tests/maintenance ?
jakub
2018-05-21 10:41
let me check manually
adan
2018-05-21 10:41
thanks
jakub
2018-05-21 10:42
hmmm
jakub
2018-05-21 10:42
it seems like some kind of race condition, I did it by hand and it fetched without issue
jakub
2018-05-21 10:42
weird
jakub
2018-05-21 10:44
fixed it
adan
2018-05-21 10:44
thanks!
adan
2018-05-21 10:44
what was the fix?
jakub
2018-05-21 10:45
`git gc --prune=now` and `git remote prune origin`
jakub
2018-05-21 10:45
I think there was a conflict in existing branches names
jakub
2018-05-21 10:45
some old branch must have had the same name as a new one, and if not name than maybe a tree reference? something like that
adan
2018-05-21 10:45
yep, this is the case
adan
2018-05-21 10:46
thank you one more time
jakub
2018-05-21 10:46
cheers!
igor680
2018-05-21 11:41
@jakub
```
FAILURE: Build failed with an exception.
* What went wrong:
A problem occurred configuring project ':app'.
> You have not accepted the license agreements of the following SDK components:
[Android SDK Build-Tools 26.0.2, Android SDK Platform 27].
Before building your project, you need to accept the license agreements and complete the installation of the missing components using the Android Studio SDK Manager.
Alternatively, to learn how to transfer the license agreements from one workstation to another, go to http://d.android.com/r/studio-ui/export-licenses.html
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
```
jakub
2018-05-21 11:42
hahahaha
igor680
2018-05-21 11:42
`./sdkmanager --licenses`
jakub
2018-05-21 11:42
oh right, I forgot to agree to give my firstborn's soul to satan
igor680
2018-05-21 11:42
lol :slightly_smiling_face:
igor680
2018-05-21 11:42
but yeah, Android SDK requires that
igor680
2018-05-21 11:43
because Google is not evil, of course
jakub
2018-05-21 11:43
didn't they just drop that motto?
igor680
2018-05-21 11:43
maybe, when started building millitary drones? :slightly_smiling_face:
jakub
2018-05-21 11:44
or maybe when they started pumping petabytes of data to NSA?
jakub
2018-05-21 11:46
`Have you read the agreement? (yes/no/WHO CARES)`
jakub
2018-05-21 11:46
done
igor680
2018-05-21 11:46
both machines? :slightly_smiling_face:
jakub
2018-05-21 11:46
yeah
igor680
2018-05-21 11:46
let me try again...
jakub
2018-05-21 11:46
but there's another issue in my nightly
jakub
2018-05-21 11:46
`NDK is missing a "platforms" directory.`
jakub
2018-05-21 11:48
found the issue, the build itself specifies bad `ANDROID_HOME` and friends
igor680
2018-05-21 11:56
whoops :slightly_smiling_face:
igor680
2018-05-21 11:57
can I restart now?
jakub
2018-05-21 11:57
sure
dmitryn
2018-05-21 12:17
what about changing ?apk/ipa uploaded? message in PRs posted by @roman to be posted by https://github.com/statustestbot ?
dmitryn
2018-05-21 12:24
Otherwise, i confused seeing Roman?s face in Chrome notification, thinking that he?s really making comment on PR.
nastya
2018-05-21 12:41
:heavy_plus_sign:
dmitryn
2018-05-21 12:57
@nastya do you know who owns https://github.com/statustestbot ? we need github token for that user
nastya
2018-05-21 12:57
@adan could you assist?
adan
2018-05-21 12:58
not me, @pedro probably could
igor680
2018-05-21 13:11
```
FAILURE: Build failed with an exception.
* What went wrong:
Execution failed for task ':app:validateSigningRelease'.
> Keystore file /Users/jenkins/.gradle/status-im.keystore not found for signing config 'release'.
```
:sadpanda:
jakub
2018-05-21 13:11
eeee, but I've put it in place
igor680
2018-05-21 13:11
but yeah, we need to copy this file from `.gradle` on `macos1`
jakub
2018-05-21 13:11
which host?
jakub
2018-05-21 13:11
I did
jakub
2018-05-21 13:11
it's there
igor680
2018-05-21 13:11
`Running on macos2 in <blah>`
jakub
2018-05-21 13:11
```
administrators-Mac-Pro:~ administrator$ ls -l /Users/jenkins/.gradle/status-im.keystore
-rw-r--r-- 1 jenkins staff 2289 May 21 13:21 /Users/jenkins/.gradle/status-im.keystore
```
jakub
2018-05-21 13:12
when did this fail?
igor680
2018-05-21 13:12
just before the town hall
jakub
2018-05-21 13:13
link?
igor680
2018-05-21 13:13
jakub
2018-05-21 13:14
Looks like a different issue
jakub
2018-05-21 13:14
`Execution failed for task ':app:transformNativeLibsWithStripDebugSymbolForRelease'.`
jakub
2018-05-21 13:14
seems like a JVM issue, give me a sec
igor680
2018-05-21 13:15
ah, that was your build, it was after mine :slightly_smiling_face:
igor680
2018-05-21 13:15
```
> A problem occurred starting process 'command '/usr/local/share/android-ndk/toolchains/mips64el-linux-android-4.9/prebuilt/darwin-x86_64/bin/mips64el-linux-android-strip''
```
igor680
2018-05-21 13:15
that is weird
roman
2018-05-21 13:16
i asked @pedro and he told tha we can?r replace my token with some app?s token
roman
2018-05-21 13:16
we probably need to create another user or so
jakub
2018-05-21 13:17
```
ICE-MIPS64
This mips64el-linux-android-4.9 directory exists to make the NDK compatible with the Android
SDK's Gradle plugin, version 3.0.1 and earlier, which expects the NDK
to have a MIPS64 toolchain directory.
```
roman
2018-05-21 13:17
also i would send one message for both builds, because in some PRs it's a bit spammy now
jakub
2018-05-21 13:17
hmmmm
jakub
2018-05-21 13:17
`bin/mips64el-linux-android-strip` doesn't exist
jakub
2018-05-21 13:18
right
jakub
2018-05-21 13:18
```
administrators-Mac-Pro:~ administrator$ ls -l /android-sdk/ndk-bundle/toolchains/mips64el-linux-android-4.9/prebuilt/darwin-x86_64/
total 240
-rwxrwxrwx 1 root wheel 18002 Nov 10 2017 COPYING
-rwxrwxrwx 1 root wheel 26527 Nov 10 2017 COPYING.LIB
-rwxrwxrwx 1 root wheel 3324 Nov 10 2017 COPYING.RUNTIME
-rwxrwxrwx 1 root wheel 35147 Nov 10 2017 COPYING3
-rwxrwxrwx 1 root wheel 7639 Nov 10 2017 COPYING3.LIB
-rwxrwxrwx 1 root wheel 0 Nov 10 2017 MODULE_LICENSE_GPL
-rwxrwxrwx 1 root wheel 18002 Nov 10 2017 NOTICE
drwxrwxrwx 29 root wheel 986 Nov 10 2017 bin
drwxrwxrwx 5 root wheel 170 Nov 10 2017 lib
drwxrwxrwx 3 root wheel 102 Nov 10 2017 libexec
drwxrwxrwx 7 root wheel 238 Nov 10 2017 mips64el-linux-android
-rwxrwxrwx 1 root wheel 1092 Nov 10 2017 repo.prop
```
jakub
2018-05-21 13:18
in macos1 it exists
igor680
2018-05-21 13:19
wrong ndk version?
jakub
2018-05-21 13:19
possibly
jakub
2018-05-21 13:20
macos1
```
$ cat source.properties
Pkg.Desc = Android NDK
Pkg.Revision = 15.2.4203891
```
jakub
2018-05-21 13:20
macos2
```
$ cat /usr/local/share/android-ndk/source.properties
Pkg.Desc = Android NDK
Pkg.Revision = 17.0.4754217
```
jakub
2018-05-21 13:21
looks like a downgrade is in order, apparently
jakub
2018-05-21 13:21
why do we use such outdated NDK though?
igor680
2018-05-21 13:21
no idea :slightly_smiling_face:
igor680
2018-05-21 13:21
I guess we need to put somewhere a list of issues to be fixed, including outdated toolkit/deps
jakub
2018-05-21 13:31
okay, I downgrade it to v16
jakub
2018-05-21 13:31
with
```
brew cask install https://raw.githubusercontent.com/caskroom/homebrew-cask/7c6b5f9c167497bf8fa0c06fd5b133c0856d639b/Casks/android-ndk.rb
```
jakub
2018-05-21 13:31
it provides the `mips64el-linux-android-4.9` toolchain
igor680
2018-05-21 13:32
okaaaay
igor680
2018-05-21 13:32
does it compile now?
jakub
2018-05-21 13:32
lets see
jakub
2018-05-21 13:33
jakub
2018-05-21 13:47
```
Failed to read key status from store "/Users/jenkins/.gradle/status-im.keystore": Keystore was tampered with, or password was incorrect
```
jakub
2018-05-21 13:48
weird, md5 sum is the same as on macos1
jakub
2018-05-21 13:48
maybe it's about the modification time...
jakub
2018-05-21 13:52
okay, i think it's about `/Users/jenkins/.gradle/gradle.properties`
jakub
2018-05-21 14:09
Looks like `jq` was missing, added that too
jakub
2018-05-21 14:09
igor680
2018-05-21 14:24
lol, we need a proper Xcode too :slightly_smiling_face:
igor680
2018-05-21 14:25
we need to convert this thread into a document when we are done
jakub
2018-05-21 14:25
I am makomgnjotes of these thinga in my todo doc
jakub
2018-05-21 14:25
I intend to make an ansible rolewhen i have time
igor680
2018-05-21 14:26
yeah! that sounds great
jakub
2018-05-21 14:26
(I hate touchscreens)
jakub
2018-05-21 14:26
Also, i dod install xcode, but maybe so.e part is missing
jakub
2018-05-21 14:26
Ill be back before 6pm and take a look
jakub
2018-05-21 14:27
(Physical keyboards ftw)
igor680
2018-05-21 14:27
yeah, maybe you need to launch and install cmd tools or something
igor680
2018-05-21 14:37
We really need to enforce timeouts on our build in Jenkins: https://jenkins.io/doc/pipeline/steps/workflow-basic-steps/#code-timeout-code-enforce-time-limit
igor680
2018-05-21 14:50
to avoid Jenkins jobs to be stuck forever, I wrapped every job in a timeout of 90 minutes: https://github.com/status-im/status-react/pull/4368
jakub
2018-05-21 17:30
when i have the time I'll automate it
arnetheduck.slack
2018-05-22 06:50
following up on https://status-im.slack.com/archives/C9N44MFAL/p1526539432000288, _why_ are we signing commits for some projects, _which_ are these projects and _who_ sets it up?
pedro
2018-05-22 07:16
the problem is that a GitHub App's token is transient (it is refreshed every hour or so using a special API). So we either need to turn that functionality into a GitHub app, or use some special user's token (like https://github.com/statustestbot)
jakub
2018-05-22 07:38
heads up, Digital Ocean is experiencing some kind of attack today:
https://i.imgur.com/KN1CZw0.png
Our droplets and the management console might be unavailable.
jakub
2018-05-22 07:39
Interestingly enough their status dashboard shows everything green: https://status.digitalocean.com/
adamb
2018-05-22 07:50
Maybe these are just deploys? :stuck_out_tongue: Experienced the same yesterday but just for a few seconds
adamb
2018-05-22 07:51
oh actually it does not work 20 mins later?
adamb
2018-05-22 07:52
but there is an incident info: https://status.digitalocean.com/incidents/17by51tngyb8
jakub
2018-05-22 07:54
Its not just the panel, terraform gets stuck too when talking to their api
adamb
2018-05-22 07:54
yeah, I guess it's the panel + api
adamb
2018-05-22 07:54
but they transparency sucks as they don't mention API
igor680
2018-05-22 08:20
@jakub re: macos2 and macos3 runners. Is there any chance I can get access to these machines? I have some time today and I really want to make them work :slightly_smiling_face:
jakub
2018-05-22 08:20
I think your key should be added to administrator aacoubt
jakub
2018-05-22 08:21
If not Adams should be, you can ask him
jakub
2018-05-22 08:22
I did run the command to enable cmd tolls for xcode, is ot still failing?
igor680
2018-05-22 08:23
I'll try to test that.
igor680
2018-05-22 08:24
Btw, I took `macos3` offline to make sure that my test PR always runs on `macos2`. As soon as that is done, I'll enable `macos3` back.
igor680
2018-05-22 08:41
still
```
[react_pull_requests_PR-4357-UEDTNANIQEEGI3UR3VPZ23GBNLNTFGI2IJLGELQNL5Q5IUPEGFUA] Running shell script
+ export RCT_NO_LAUNCH_PACKAGER=true
+ RCT_NO_LAUNCH_PACKAGER=true
+ xcodebuild -workspace ios/StatusIm.xcworkspace -scheme StatusIm -configuration release -archivePath status clean archive
xcode-select: error: tool 'xcodebuild' requires Xcode, but active developer directory '/Library/Developer/CommandLineTools' is a command line tools instance
```
igor680
2018-05-22 08:41
probably you need to run xcode-select manually before all of that
igor680
2018-05-22 08:42
this looks like exactly the issue: https://github.com/nodejs/node-gyp/issues/569
igor680
2018-05-22 08:44
jakub
2018-05-22 08:44
I did run that command in that issue on both hosts
jakub
2018-05-22 08:47
(Btw can you ssh into the hosts?)
jakub
2018-05-22 08:54
Makes sense
igor680
2018-05-22 09:21
Let me try...
igor680
2018-05-22 09:23
@jakub it asks me for a password, so I guess I can't :slightly_smiling_face:
jakub
2018-05-22 09:30
Are you using the `administrator` user? If so please ask Adam to add your key, sorry I didnt do it from the start
igor680
2018-05-22 09:36
tried the administrator user, asks for a password anyway
igor680
2018-05-22 09:36
@adamb can you add my ssh key on `macos2` and `macos3` machines?
adamb
2018-05-22 09:45
I can't ssh. It asks me for a pass for `adamb` and `administrator` users
jakub
2018-05-22 09:45
Well damn, i suck then, let me forward emails from MacStadium to you
jakub
2018-05-22 09:46
Actually i do t have to
jakub
2018-05-22 09:47
You have them in pass
jakub
2018-05-22 09:47
Under hosts
adamb
2018-05-22 09:48
oh nice
igor680
2018-05-22 09:48
LastPass?
jakub
2018-05-22 09:48
No infra-pass repo
adamb
2018-05-22 09:51
@igor680 can you please send a PR with your public key here https://github.com/status-im/infra-role-bootstrap/tree/master/files?
igor680
2018-05-22 10:04
Ah, gotcha! Will do today!
goranjovic
2018-05-22 14:24
Why does this PR say that it both can and can't be built? https://github.com/status-im/status-react/pull/4194
jeluard
2018-05-22 14:27
We just switched to quantum computers
goranjovic
2018-05-22 14:30
Ah yeah, I just need to open the box and let the cat out :joy:
igor680
2018-05-22 14:31
timeout?
goranjovic
2018-05-22 14:58
Looks like there is an extra check that's not present on other PRs and it's failing. @maxr Do you have a clue what's going on? Asking you because there's your picture on it :slightly_smiling_face:
roman
2018-05-22 15:17
we also need to figure finally how to abort simultaneous builds for the same branch
igor680
2018-05-22 15:18
yeah, I did some googling, it isn't super-obvious (not just a checkbox)
maxr
2018-05-22 16:10
@goranjovic I will check in 1 hour ... I've added new pipeline for desktop, but as I've believed it should be triggered if new added Jenkins.desktopbuild file is found, but it seems branch filtering should be updated as well, sorry, about that
maxr
2018-05-22 16:17
@goranjovic if that check failure is dated somewhere like last Sunday, could, you please re-trigger Jenkins checks again ... it shouldn't appear any more
maxr
2018-05-22 17:29
Hey @goranjovic it seems you can just ignore this Check failure and merge your PR (since the merge button is active for me) ... it shouldn't appear on next PRs, hopefully :slightly_smiling_face:
dmitryn
2018-05-22 17:55
@pedro not sure we need GH App for that, we just need `statustestbot` user API token with `public_repo` scope to make comments https://github.com/status-im/status-react/blob/develop/Jenkinsfile#L69
goranjovic
2018-05-23 06:28
@maxr No worries. Seems to be fine now that I re-triggered it by pushing to the branch again. Just re-running on jenkins wasn't enough. Thanks!
jakub
2018-05-23 07:29
finally managed to get DigitalOcean to increase our droplet limit to a 100 yesterday
sergii
2018-05-23 10:43
Hi! by some reason latest nightly iOS app is gone from diawii for me :disappointed: It was there 1h ago for sure, but now it `..is not available anymore` https://status-im.github.io/nightly/
arnetheduck.slack
2018-05-24 09:57
jakub
2018-05-24 12:29
hey, do we have a company apple account or something like that? I can't download xcode for new slaves without logging with an apple account
naghdy
2018-05-24 12:32
Not that i know. cc @carl
igor680
2018-05-24 12:42
Anyone with iTunes Connect access can do that
jakub
2018-05-24 12:42
nah its fine, i think I managed without it(we'll see)
nastya
2018-05-25 14:51
@jakub nightly build is failing, is this something u can assist with? https://jenkins.status.im/job/status-react/job/nightly/406/
jakub
2018-05-25 14:53
Yeah, just merged a fix
nastya
2018-05-28 09:38
@jakub `[nightly] Running shell script
+ rm -rf node_modules
rm: node_modules: Directory not empty` when triggering nightly
igor680
2018-05-28 09:41
can you send me the link to the job?
nastya
2018-05-28 10:06
@igor680 any insight?
igor680
2018-05-28 10:07
nope, I restarted it let's see if that issue is permanent
igor680
2018-05-28 10:19
@nastya seems to just work now...
nastya
2018-05-28 10:19
:+1:
nastya
2018-05-28 10:20
waiting then, thanks
jakub
2018-05-28 10:21
I think it happened because I ran a build manually from Xcode on the host
jakub
2018-05-28 10:21
not sure why `rm -fr` would fail considering the `-f`...
cryptowanderer
2018-05-28 20:06
@jakub https://blog.status.im/ <- is done for me with a `too many redirects` error. Any idea why?
jakub
2018-05-28 21:16
Yeah, i have an idea what it might be, i can fix it in ~2h
jakub
2018-05-28 21:16
Actually less than that
jakub
2018-05-28 22:53
Works now
sergii
2018-05-29 08:39
@jakub e2e tests job hang up from yesterday on one of the PR. I?m going to stop the last job (pushed against PR-4499) https://jenkins.status.im/job/end-to-end-tests/job/status-app-end-to-end-tests/
jakub
2018-05-29 08:40
Sure, go ahead
igor680
2018-05-29 09:06
@jakub I need your thoughts or approval stamp here :slightly_smiling_face:
https://github.com/status-im/status-react/pull/4507
jakub
2018-05-29 09:06
checking now
jakub
2018-05-29 09:07
I see
igor680
2018-05-29 09:08
thanks!
jacqueswww
2018-05-29 09:57
Hey guys, still wanted to add my thoughts to the blocked bootnode discussion. We can learn a *a lot* here from how tor works.
jacqueswww
2018-05-29 09:58
Basically there is the concept of Bridges and obfuscated bridges.
jacqueswww
2018-05-29 10:00
This is how one would effectively counter-attack a mass attack against ones' IP addresses. One would definitely have a certain amount of 'hidden' addresses that get obfuscated and only revealed to those that really need it. As can be seen above.
jacqueswww
2018-05-29 10:06
Also if you need more IPs in close by countries, lowendtalk & lowendbox is great place to find them :wink: https://lowendbox.com/
divan
2018-05-29 10:20
thanks, never heard of. what is this community is all about?
jacqueswww
2018-05-29 10:23
It's basically dedicated to lowend / cheap VPS's
jacqueswww
2018-05-29 10:23
:slightly_smiling_face:
jacqueswww
2018-05-29 10:23
But it's a great place to find Linux boxen in less common areas
jacqueswww
2018-05-29 10:24
Because if you look at the big picture, most IPs are like in a handful main routes. DO/Amazon etc. but using the forums you get access to all different datacenters and routes.
jacqueswww
2018-05-29 10:25
Also useful if we want to keep completely dis-associated from the servers.
dmitry.shulyak
2018-05-29 10:56
so in the essence those are nothing more than addresses that are not distributed with tor/status app. and in our case those will be enodes for bootnodes that user adds manually
dmitry.shulyak
2018-05-29 10:57
the problem with this approach is that if you really want to - you can monitor dht table and those bootnodes will be there
jacqueswww
2018-05-29 11:05
sure, dht makes it more difficult :confused:
jacqueswww
2018-05-29 11:05
but you can make that list adjustable/dynamic vs. the hard coded list you publish with the app.
dmitry.shulyak
2018-05-29 11:07
:+1: that for sure would be harder to ban. btw we will allow to set bootnodes using status app in beta
jacqueswww
2018-05-29 11:07
cool
dmitry.shulyak
2018-05-29 11:07
but there is no channel to distribute such bootnodes yet
jacqueswww
2018-05-29 11:08
yeah tricky, guess we could just periodically use pastebin + gist.github ?
jacqueswww
2018-05-29 11:09
Then encode the url in some DNS txt record?
jacqueswww
2018-05-29 11:10
trying to think of the most obfuscated way to do this :stuck_out_tongue:
dmitry.shulyak
2018-05-29 11:11
who will resolve them then?
jacqueswww
2018-05-29 11:12
the status app
jacqueswww
2018-05-29 11:12
we could take it even further and have some time-based dns lookup ?
jacqueswww
2018-05-29 11:14
Another trick I know
jacqueswww
2018-05-29 11:14
I using ICMP/ping packets
dmitry.shulyak
2018-05-29 11:14
i meant where that mapping will be stored. but i didn't knew that it is possible to associate dns name with arbitrary txt record
jacqueswww
2018-05-29 11:16
Another option I just thought of... take a deleted wikipedia article.
jacqueswww
2018-05-29 11:16
and hide it in the comments or something :wink:
jacqueswww
2018-05-29 11:16
I don't think russia will ever really block wikipedia?
jacqueswww
2018-05-29 11:19
still might be an option :stuck_out_tongue:
jacqueswww
2018-05-29 11:26
An IP address is also data
jacqueswww
2018-05-29 11:26
So you can encode it that way as well... in an dns entry
jacqueswww
2018-05-29 11:27
:grin:
vitaliy
2018-05-29 14:06
hi guys, can you please help me with GitHub/Jenkins Slack integrations? I'd like to setup the following:
- send notifications about Jenkins to a separate Slack channel (e.g. ?desktop-jenkins?)
- send notifications about GitHub commits to a specific repo/branch/labelled PRs to a separate Slack channel (e.g. ?desktop-github?)
Not sure where to start with these.
maxr
2018-05-29 14:18
Hello there,
I'm trying to do `npm install` for status-react desktop on linux1 Jenkins agent machine, but npm hangs for me.
Currently installed node version - v9.11.1, npm version - 6.0.1
Wondering if it will be fine to downgrade node version to Node v8 LTS
vitaliy
2018-05-29 14:29
there have been some issues with npm 6.0.1 and 6.1.0, at least for me downgrading to 5.5.1 always helped. Not sure if 5.5.1 is supported by Node 9 though
rob018
2018-05-29 16:10
@maxr I was running 6.0.1 on linux and experienced the same hanging behavior. downgrading to 5.5.1 helped as vitaliy mentioned
rob018
2018-05-29 16:15
Hey all, we've been receiving some jenkins errors related to permissions on open bounty. https://jenkins.status.im/blue/organizations/jenkins/status-openbounty%2Fstatus-openbounty-app/detail/PR-460/2/pipeline. Continuing to look into it but wanted to share in case anyone has experience with this.
maxr
2018-05-29 17:09
@vitaliy @rob018 npm@5.5.1 works finally, thanks!
rob018
2018-05-29 17:11
nice!
divan
2018-05-30 10:25
Awesome, thanks for sharing :slightly_smiling_face:
divan
2018-05-30 14:05
security through obscurity is dubious apporach :slightly_smiling_face:
divan
2018-05-30 14:39
@jacqueswww what is time based dns lookup btw?
jacqueswww
2018-05-30 14:57
Well the idea is one selects let's say 100 domains.
And the based on the current time in UTC one of those domains will respond with the correct information. In a ~6h window or the like.
divan
2018-05-30 15:24
But the adversary (or whoever is blocking internet) will be able to inspect source code, and resolve all 100 domains all the time, and block IP addresses it responds to, no?
jacqueswww
2018-05-30 15:48
Sure, but has to script it actively
jacqueswww
2018-05-30 15:49
And you only make the IPs available on 1 of the 100 domains for a given timeframe.
jacqueswww
2018-05-30 15:50
Allowing you to change the emitted IPs actively, but also make firewalling it considerbly difficult.
jacqueswww
2018-05-30 15:51
The biggest problem is that DNS is just as vulnerable... There is no easy fix.:no_mouth:
divan
2018-05-30 16:41
My thinking goes into assumption that there is no 100% safe way to overcome blocking, and we're inevitably end up having arms race with watchdog.
And that's ok. We just need to make blocking us a lot more expensive operation that launching and advertising new bootnode.
I.e. make it "single click" action for more or less technical people. For example, we're launched today new bootnode and it took quite a lot of efforts and our dedicated devops person. It should be as easy as `docker run ...` instead.
We may even work on custom hardware and give it for free/at cost price ? it's essentially should be just a router/access point with discovery service running on it.
anna
2018-06-01 08:19
morning :slightly_smiling_face:
anna
2018-06-01 08:22
not sure how it can be - nightly build on Jun 1, 04:59 failed https://jenkins.status.im/job/status-react/job/nightly/423/ but https://status-im.github.io/nightly/ shows that it was updated on 2018/06/01 02:42:09 UTC and successful build is shown in https://status-im.slack.com/archives/C8RNGLKRT/p1527821287000157 Is there other job for nightly?
nastya
2018-06-01 08:24
@anna ^ is yesterday's nighlty
nastya
2018-06-01 08:24
however we need to do smth with this `updated` :slightly_smiling_face:
anna
2018-06-01 08:24
ok, so updated is wrong?
nastya
2018-06-01 08:24
yeah seems it recorders an attempt to build a build ) but does not care about the outcome (success or fail)
nastya
2018-06-01 08:25
but @jakub may be knows more
anna
2018-06-01 08:26
but what about jenkins channel, how this was built? https://status-im.slack.com/archives/C8RNGLKRT/p1527821287000157
jakub
2018-06-01 08:27
Oh, it failed and it was updated anyway?
anna
2018-06-01 08:27
not sure yet that it failed, please read my message and look at the links
anna
2018-06-01 08:27
there is something that made a build of nightly (or uses same title) but it's not this job https://jenkins.status.im/job/status-react/job/nightly/
anna
2018-06-01 08:28
@adan @levgeniia maybe it's some new test jobs?
anna
2018-06-01 08:33
So, https://status-im.github.io/nightly/ is updated with links from https://status-im.slack.com/archives/C8RNGLKRT/p1527821287000157 but I still do not understand how this build was done if there was no successful nightly builds on June 1 in https://jenkins.status.im/job/status-react/job/nightly/
adan
2018-06-01 08:33
nope, this is not related to tests
adan
2018-06-01 08:33
maybe nightly-release?
adan
2018-06-01 08:34
ignore this
anna
2018-06-01 08:35
ok, just for other readers: nightly release creates other apk name (with ?rel?) like im.status.ethereum-68eb36-rel.apk
anna
2018-06-01 08:35
so it's not nightly release
anna
2018-06-01 08:39
only 1 parametrized job was done on June 1 https://jenkins.status.im/job/status-react/job/parametrized_build/656/ and ios link in that job is not same as in the nightly: ARTIFACT iOS: https://i.diawi.com/5LvMYQ
anna
2018-06-01 08:39
so, I don't think it's param build
adan
2018-06-01 08:40
okay in https://jenkins.status.im/job/status-react/job/status-nightly-publish-link/162/ I can see that the job has been started by nightly-tests job
adan
2018-06-01 08:40
which is very strange
adan
2018-06-01 08:41
yes
adan
2018-06-01 08:42
tests != e2e tests
anna
2018-06-01 08:42
maybe infra team has other tests to try?
igor680
2018-06-01 08:42
@jakub btw, did you manage to enable other mac runners? :slightly_smiling_face:
jakub
2018-06-01 08:42
no
jakub
2018-06-01 08:43
I focused on the russia stuff, scaling and logging based on what the top priorities were
jakub
2018-06-01 08:43
This Xcode signing stuff is insane
jakub
2018-06-01 08:43
I have no idea how to make it work
jakub
2018-06-01 08:43
I have added the iTunes account, imported the profile, imported the certificate
jakub
2018-06-01 08:44
doesn't work
igor680
2018-06-01 08:44
okaaay
I might take a look next week then
jakub
2018-06-01 08:44
I can work on it a bit more but the top priority right now is multi-dc scaling as far as I can tell
igor680
2018-06-01 08:45
> Xcode signing stuff is insane
Yeah, it always have been
jakub
2018-06-01 08:45
yeah, maybe you can notice something I didn't
anna
2018-06-01 08:45
@jakub I see some of the builds were started by you, so maybe you can tell what are these tests about? (e.g. this job https://jenkins.status.im/job/tests/job/nightly-test/46/)
jakub
2018-06-01 08:45
(also, the VNC sessions are so goddamn slow it makes me want to kill myself)
jakub
2018-06-01 08:46
oh, this build isn't disabled?
jakub
2018-06-01 08:46
damn, then that explains it I guess
jakub
2018-06-01 08:47
that was intended for running builds on the new jenkins slaves, but I guess I forgot to disable it tirggering on new commits
igor680
2018-06-01 08:54
:sadpanda:
nastya
2018-06-01 19:38
@jakub can't download http://artifacts.status.im:8081/artifactory/nightlies-local/im.status.ethereum-b47caf-n-fl.apk - page just spinnes for ages. It is from https://status-im.github.io/nightly/
nastya
2018-06-01 19:46
seems http://139.162.11.12:8081/artifactory/nightlies-local/ is taking a weekend break
maxr
2018-06-01 19:49
had the same issue few hours ago, was not able to download it finally
jakub
2018-06-02 22:52
The cobtainer needs a restart, i cant do it, away from city and laptop, @adamb should be able to if its important
jakub
2018-06-02 22:52
Its on my neverending todo list =]
nastya
2018-06-04 07:22
@adamb @jakub nightly is still broken (
jacqueswww
2018-06-04 12:36
@jakub just checked out grafana. Looking Good! :rocket:
jakub
2018-06-04 12:37
thanks man, monitoring is crucial for making this scalable, once I finish the multi-datacenter setup for eth-cluster I can configure log aggregation as well
jacqueswww
2018-06-04 12:37
awesome ! :slightly_smiling_face:
jakub
2018-06-04 12:37
sorry for not being available for my slides but zoom app on linux seems to be really unstable
jacqueswww
2018-06-04 12:37
np
jacqueswww
2018-06-04 12:37
I am a big graphite/grafana fan
jacqueswww
2018-06-04 12:38
Which storage engine does this setup use?
jakub
2018-06-04 12:38
well, this is prometheus/grafana, because graphite uses a push schema, and prometheus is using pull, which leverages our Consul host/service inventory
hutch
2018-06-05 05:23
Kudos @jakub @oskarth @adamb and everyone else that worked so hard to make these latest builds mailbox nodes etc. reliable and demonstrate the full potential and future of status. Bravo :clap:
pedro
2018-06-05 16:16
Anyone here versed in maven who can help me understand what's happening with a Jenkins build? I'm trying to build a status-react branch with a new status-go dependency, and it is saying it cannot find it (but it is clearly a local issue, since I was able to build on another machine) https://jenkins.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-4642/3/console
adamb
2018-06-06 09:00
Is it true that we use Artifactory and all this maven crap just for status-go?
adamb
2018-06-06 09:01
Btw. this PR build pass
pedro
2018-06-06 09:10
yeah, it passed after a certain timeout period after which it tries to download the artifact again
roman
2018-06-11 15:33
i can't connect to 207.254.71.75 using Screen sharing (i need this for adding new UDID), can someone restart service from admin user on this machine?
```
sudo launchctl unload /System/Library/LaunchDaemons/com.apple.screensharing.plist
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.screensharing.plist
```
or just add my key so i can use ssh and do this
Also we probably need to figure how to update profiles on build machine automatically (when we add new device to ad hoc profile)
roman
2018-06-12 08:51
@jakub @adamb
jakub
2018-06-12 08:52
sure
jakub
2018-06-12 08:52
done
roman
2018-06-12 08:55
did you change password for jenkins user?
jakub
2018-06-12 08:55
nope
roman
2018-06-12 08:55
the old one doesn?t work anymore, weird
jakub
2018-06-12 08:55
what's your key?
roman
2018-06-12 08:56
key? i need ui thus i need password in order to login through ui
jakub
2018-06-12 08:56
oh, you're talking about the password to the jenkins ui?
jakub
2018-06-12 08:56
but isn't login managed by googlle oauth?
roman
2018-06-12 08:57
ok so? we have that macstadium instance. There are two users on this macos machine: administartor and jenkins
jakub
2018-06-12 08:57
mhm
roman
2018-06-12 08:58
so i need access to this jenkins user
roman
2018-06-12 08:58
in UI
jakub
2018-06-12 08:58
why?
roman
2018-06-12 08:58
because i have no idea how to update profiles otherwise
roman
2018-06-12 08:58
please read my original message
roman
2018-06-12 08:59
> Also we probably need to figure how to update profiles on build machine automatically (when we add new device to ad hoc profile)
jakub
2018-06-12 08:59
what profiles are you talking about?
jakub
2018-06-12 08:59
in xcode?
roman
2018-06-12 08:59
yes
jakub
2018-06-12 08:59
hmmm, they wouldn't you need access to the UI via VNC?
jakub
2018-06-12 09:00
or can you do this in CLI?
jakub
2018-06-12 09:00
if so, then please give me your public ssh key, I'll add it to jenkins user `authorized_keys`
roman
2018-06-12 09:01
ok, my key is already there
roman
2018-06-12 09:01
i don't know how to do this through cli
jakub
2018-06-12 09:01
ok, then you'll need to VNC into the machine
jakub
2018-06-12 09:01
i think the password is in lastpass
jakub
2018-06-12 09:01
give me a sec
roman
2018-06-12 09:01
yes correct, and i have vlc
roman
2018-06-12 09:01
it works
jakub
2018-06-12 09:02
oh, you do?
roman
2018-06-12 09:02
i only don't know password for jenkins user
roman
2018-06-12 09:02
because someone changed it
roman
2018-06-12 09:02
whatever
roman
2018-06-12 09:02
i will just change it now
jakub
2018-06-12 09:02
hmmm, I never used it
jakub
2018-06-12 09:02
I always ssh as administrator and then sudo into jenkins
jakub
2018-06-12 09:02
I didn't know there even was a password
roman
2018-06-12 09:03
i?m not saying you did this, but it is changed
roman
2018-06-12 09:03
�\_(?)_/�
jakub
2018-06-12 09:03
yeah yeahj
roman
2018-06-12 09:03
i used it like for year or so
jakub
2018-06-12 09:03
change it back then
roman
2018-06-12 09:06
ok, done
roman
2018-06-12 09:07
anyway, we need to figure how to add new devices without this vlc ritual
jakub
2018-06-12 09:07
I don't know if there is a way, Apple doesn't like cli
jakub
2018-06-12 09:07
it would be great though
jakub
2018-06-12 09:07
Think Different?
anna
2018-06-12 15:16
anna
2018-06-12 15:16
``` [react_pull_requests_PR-4727-4SWQJSKPLTVPMUEXEQWQADYJNQIZCTJVQXFLWVELEM7RMG46CITQ] Running shell script
+ curl https://upload.diawi.com/ -F token=**** -F file=@android/app/build/outputs/apk/release/app-release.apk -F find_by_udid=0 -F wall_of_apps=0
+ jq -r .job
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
45 50.3M 0 0 45 23.0M 0 21.9M 0:00:02 0:00:01 0:00:01 21.9M
100 50.3M 0 13 100 50.3M 6 23.5M 0:00:02 0:00:02 --:--:-- 23.5M
100 50.3M 0 13 100 50.3M 6 23.5M 0:00:02 0:00:02 --:--:-- 23.5M
parse error: Invalid numeric literal at line 1, column 8 ```
anna
2018-06-12 15:18
@jakub can you have a look at this failure, please? I've restarted the build but not sure what are the recent changes in our builds
jakub
2018-06-12 15:18
sure, give me a 3min and I'll check it out
jakub
2018-06-12 15:26
okay, so this is a `jq` parsing error, which means that the JSON it got from the curl command was fauly
jakub
2018-06-12 15:28
too bad we don't save that output
jakub
2018-06-12 15:35
interestingly enough this time it worked fine
anna
2018-06-12 15:35
I was not patient enough :slightly_smiling_face:
jakub
2018-06-12 15:35
if this happens again it might be worth adding `tee` before that pipe to `jq` and try to save the output to a file
jakub
2018-06-12 15:36
tell me if you see that error again
nastya
2018-06-12 20:46
/kudos @jakub for fixing nightly!
jakub
2018-06-12 20:47
my face when an empty commit fixes artifactory uploads
igor680
2018-06-14 08:09
@jakub :rotating_light: JENKINS :rotating_light:
```
java.io.IOException: No space left on device
at sun.nio.ch.FileDispatcherImpl.write0(Native Method)
at sun.nio.ch.FileDispatcherImpl.write(FileDispatcherImpl.java:60)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:93)
at sun.nio.ch.IOUtil.write(IOUtil.java:65)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:211)
at hudson.util.FileChannelWriter.write(FileChannelWriter.java:72)
at java.io.Writer.write(Writer.java:192)
at hudson.util.AtomicFileWriter.write(AtomicFileWriter.java:162)
at java.io.Writer.write(Writer.java:157)
at hudson.XmlFile.write(XmlFile.java:189)
at hudson.model.User.save(User.java:827)
```
jakub
2018-06-14 08:09
oh snap
jakub
2018-06-14 08:10
slave or master?
igor680
2018-06-14 08:10
probably master, I can't even open it
jakub
2018-06-14 08:10
ah, master
jakub
2018-06-14 08:15
should be fine now
igor680
2018-06-14 08:15
yup, seems so
jakub
2018-06-14 08:15
dropped some old backups of jenkins and some build history
jakub
2018-06-14 08:15
this is why hosts need to be attached to monitoring
jakub
2018-06-14 08:16
now that logging aggregation is done I can work on other things, I'll probably re-create the jenkins host using the new setup
jakub
2018-06-14 08:16
(saving all the jobs and builds of course)
igor680
2018-06-14 08:16
yeah, makes sense
cryptowanderer
2018-06-15 13:54
Request from the Aragon team, who are really struggling with devops and infra (btw @jakub if you know someone really good, they are desperate to find people)
```
wondering if we can piggy back off of your infra somehow
devops is like the bane of our existence; something I didn't mention yesterday was how fragile these nodes seem to be
our geth node (there were some problems in the last few versions) randomly gets really slow at syncing, so we find out that it's a few thousand blocks behind when people start to complain
now our parity node earlier today was having issues
not to mention we've actually lost data in ipfs because our node pinning stuff went down one time...
I have no idea what the cool kids are using to monitor their ETH / IPFS nodes, or if any good software even exists for that. If not, would be really, really nice to have
```
jakub
2018-06-15 13:55
we use our own software for monitoring `geth`
jakub
2018-06-15 13:56
and about good devops people, I can ask around, but they are usually already pretty busy
jakub
2018-06-15 13:56
I know 2 people that MIGHT be interested, if you can give me some contact to the Aragon team HR or something like that I can let them know
jakub
2018-06-15 13:57
(although it would not be full time)
jakub
2018-06-15 13:57
(just to help them out with the current issues they are having)
adamb
2018-06-15 15:04
we may be challenged by these issues when we decide to run LES servers :confused:
cryptowanderer
2018-06-15 16:28
@jakub can I give Brett (one of their engineers) your email to ask questions seeing as our infra repos are private?
jakub
2018-06-15 16:29
sure
nastya
2018-06-18 08:10
nastya
2018-06-18 08:11
also nightly eded up with FAILURE: Build failed with an exception.
* What went wrong:
Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed)
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.
* Get more help at h
jakub
2018-06-18 08:21
eeeh? the hell is this fresh maddness
nastya
2018-06-18 08:35
well just letting you know
jakub
2018-06-18 08:36
never seen one like this, I see the previous one was different, thought in the same stage
nastya
2018-06-18 08:36
same jenkins different day
nastya
2018-06-18 08:36
:joy:
jakub
2018-06-18 08:36
:smile:
jakub
2018-06-18 09:04
jakub
2018-06-18 09:04
it's magic, as always
nastya
2018-06-18 13:49
it shows now expired build
nastya
2018-06-18 13:50
as well as the build should not be expired (quite fresh)
jakub
2018-06-18 13:50
not sure how this works, let me research
nastya
2018-06-18 13:50
just curious - we have some nitghtly builded - should it propagate there? i think yes
jakub
2018-06-18 13:51
`im.status.ethereum-22cb18-rel.apk` is this the build we want?
jakub
2018-06-18 13:51
or `im.status.ethereum-b40b04-n-fl.apk`?
nastya
2018-06-18 13:52
latest builded i guess
jakub
2018-06-18 13:53
looks correct to me
jakub
2018-06-18 13:53
nastya
2018-06-18 13:53
i think so , but iOS is expired not apk
jakub
2018-06-18 13:54
hmm
jakub
2018-06-18 13:54
right, but it seems like upload worked
nastya
2018-06-18 13:56
yeah
jakub
2018-06-18 13:57
looks like upload worked fine:
```
+ curl https://upload.diawi.com/ -F token=**** -F file=@android/app/build/outputs/apk/release/app-release.apk -F find_by_udid=0 -F wall_of_apps=0
+ jq -r .job
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
76 50.4M 0 0 76 38.3M 0 39.1M 0:00:01 --:--:-- 0:00:01 39.1M
100 50.4M 0 52 100 50.4M 36 35.8M 0:00:01 0:00:01 --:--:-- 35.8M
```
I'm not sure what updated the http://github.io page
nastya
2018-06-18 13:57
can we update somehow this page with nightlies/
jakub
2018-06-18 13:57
don't know, as I've said, I never worked on this
jakub
2018-06-18 13:57
I'm trying to find the repo
nastya
2018-06-18 13:57
:okay:
jakub
2018-06-18 13:59
I can't find it in our github, maybe it's private
jakub
2018-06-18 14:02
Looks like there's a build for that
nastya
2018-06-18 14:02
hm
jakub
2018-06-18 14:03
right, #454 nightly scheduled #18:
```
[Pipeline] build (Building status-react � status-nightly-publish-link)
Scheduling project: status-react � status-nightly-publish-link
Starting building: status-react � status-nightly-publish-link #181
[Pipeline] }
```
jakub
2018-06-18 14:03
jakub
2018-06-18 14:04
```
stage('Run status-nightly-publish-link job') {
build job: 'status-react/status-nightly-publish-link', parameters: [[$class: 'StringParameterValue', name: 'APK_URL', value: apkUrl], [$class: 'StringParameterValue', name: 'IOS_URL', value: ipaUrl]]
}
```
jakub
2018-06-18 14:05
okay, so the updated worked correct
nastya
2018-06-18 14:06
:morty:
nastya
2018-06-18 14:06
it was there for a couple of hours ago
jakub
2018-06-18 14:06
do you have anyone that has credentials for diawi?
nastya
2018-06-18 14:07
@roman maybe?
jakub
2018-06-18 14:08
doesn't seem to be around
nastya
2018-06-18 14:09
(((
jakub
2018-06-18 14:15
I can't do anything unless I can access Diawi, it seems like the issue is on their side
nastya
2018-06-18 14:15
thanks for looking into this
jakub
2018-06-18 14:15
the simplest solution might be to just run another nightly build
nastya
2018-06-18 14:16
can you do it please?
jakub
2018-06-18 14:18
sure
nastya
2018-06-18 14:20
:cookie:
jakub
2018-06-18 14:20
a cookie isn't enough, I'm pretty damned hungry
nastya
2018-06-18 14:21
:doughnut:!
jakub
2018-06-18 14:21
not a fan of sweets, do you have steak?
nastya
2018-06-18 14:21
want that one too :slightly_smiling_face:
jakub
2018-06-18 14:22
mmmmm, steak...
adamb
2018-06-20 08:04
I can't edit my profile on discuss. I am getting 404 :disappointed: cc @jakub
jakub
2018-06-20 08:04
sure, i'll check it out in a moment
jakub
2018-06-20 08:11
are you talking about `https://discuss.status.im/u/${USER}/preferences/account`
jakub
2018-06-20 08:11
or which one?
adamb
2018-06-20 08:11
exactly
adamb
2018-06-20 08:11
it redirects me to 404
jakub
2018-06-20 08:12
when you go to it, or when you click save?
adamb
2018-06-20 08:12
when I try to open it
jakub
2018-06-20 08:12
okay
jakub
2018-06-20 08:12
and your login is?
adamb
2018-06-20 08:12
I logged using my email `` but it displays my name as `adam`
jakub
2018-06-20 08:13
okay
jakub
2018-06-20 08:13
looks like a known issue
adamb
2018-06-20 08:14
so they recommend `bundle exec rake db:migrate`
adamb
2018-06-20 08:15
which seems dangerous :smile:
jakub
2018-06-20 08:15
yeah, I will have to stop discuss page for a while to do this
jakub
2018-06-20 08:15
but it's dodgy
jakub
2018-06-20 08:15
I want to research a bit more first
adamb
2018-06-22 10:46
Hey @jakub. Any luck with this one? It?s getting worse :smile: https://discuss.status.im/u/adam has completely different name than mine
jakub
2018-06-22 10:46
oh damn, I kinda give that one a lower priority
jakub
2018-06-22 10:46
I'll check it out today, sorry my dude
adamb
2018-06-22 10:46
no worries!
jakub
2018-06-22 11:37
hey my dude, can you check now?
adamb
2018-06-22 12:13
still the same :disappointed:
hutch
2018-06-22 12:13
is there a way to let contributors have access to our jenkins logs?
@jakub asking for a non-core contributor that has been a big help.
jakub
2018-06-22 12:45
a shame...
jakub
2018-06-22 12:46
not really, but this has been requested before, I can look into it
jakub
2018-06-22 12:47
though are we sure making jenkins build logs public is a good idea? or would we want this to have a more complex permissions model?
hutch
2018-06-22 12:48
:woman-shrugging:
hutch
2018-06-22 12:48
What are the risks?
jakub
2018-06-22 12:49
don't know, in theory all secrets should be obscured
jakub
2018-06-22 12:49
that is, if they come form jenkins credentials
jakub
2018-06-22 12:49
if we have anything sensitive that is used in jenkins builds and might not be obscured that might be dangerous
jakub
2018-06-22 12:49
but I'm just speculating
jakub
2018-06-22 12:50
is there a specific Jenkins Job that we want to have public logs for, or do you want this for all kinds of jobs in Jenkins?
hutch
2018-06-22 12:55
I can find out more specifically what the ask was.
jakub
2018-06-22 12:56
please, if you could
hutch
2018-06-22 12:58
Thanks!
jakub
2018-06-22 14:02
okay, access to jenkins builds should be read-only for anonymous users now, the links from github to Jenkins Blue Ocean UI should work for people outside of Status
jakub
2018-06-22 14:02
can you verify that with the non-core dev that you were mentioning?
rob018
2018-07-05 18:34
getting some permissions errors on today's open bounty jenkins builds. lmk @jakub if there is a good place to post the whole error message/draft it up into an issue :pray:
jakub
2018-07-05 18:47
not reallyt
jakub
2018-07-05 18:47
which test?
rob018
2018-07-05 18:48
```
Scripts not permitted to use method org.jenkinsci.plugins.workflow.support.steps.build.RunWrapper getRawBuild. Administrators can decide whether to approve or reject this signature.
``` on https://jenkins.status.im/blue/organizations/jenkins/status-openbounty%2Fstatus-openbounty-app/detail/PR-469/1/pipeline#log-1076
jakub
2018-07-05 18:49
ah
jakub
2018-07-05 18:50
approved
jakub
2018-07-05 18:50
it should work now
rob018
2018-07-05 18:50
thank you!
rob018
2018-07-05 18:50
much appreciated
sergii
2018-07-06 14:33
Hey @jakub have you seen this before `status-react � status-nightly-publish-link #201 completed with status FAILURE (propagate: false to ignore)` error while build job execution?
We pushed nightly build recently that failed with this error (on a `Run status-nightly-publish-link job` step)
adamb
2018-07-06 14:33
https://www.fwd.cloud/commit/post/drone-ios-deployments/ ? building iOS app on http://drone.io
jakub
2018-07-06 14:34
Not really, I'll check it put once I'm back in 1h, please link the build
sergii
2018-07-06 14:47
Build: https://jenkins.status.im/job/status-react/job/nightly/481/
Okay, meanwhile let me push it once again :slightly_smiling_face:
jakub
2018-07-06 16:24
Oh I see
jakub
2018-07-06 16:24
the `status-nightly-publish-link` failed
jakub
2018-07-06 16:25
```stderr: remote: Permission to status-im/status-im.github.io.git denied to status-im-auto.
```
jakub
2018-07-06 16:25
ohhh, I get it now
jakub
2018-07-06 16:25
I updated the token for github, and the user `status-im-auto` doesn't have access to that repo
jakub
2018-07-06 16:25
let me fix that
jakub
2018-07-06 16:28
done
jakub
2018-07-06 16:28
nightly should work now
sergii
2018-07-06 16:30
Brilliant! Let me push the job now?
sergii
2018-07-06 17:25
Yep, all good! https://status-im.slack.com/archives/C8RNGLKRT/p1530896912000071
Thanks a lot @jakub!
jakub
2018-07-06 17:26
cheers!
jakub
2018-07-08 21:13
yeah, I'm switching some things around
jakub
2018-07-08 21:13
5
jakub
2018-07-08 21:13
5 min and its back
yenda
2018-07-08 21:13
Ok thanks
jakub
2018-07-08 21:24
maybe a bit longer, the container is still rebuilding
jakub
2018-07-08 21:38
aaaand it's back
jakub
2018-07-08 21:38
sorry for the delay, had to fix a permissions issue
tatiana
2018-07-10 14:36
@jakub testing on desktop is blocked, because there is no free space on linux1 node (all PR builds failed). Can you please clean it up?
jakub
2018-07-10 14:36
sure
jakub
2018-07-10 14:42
should be fine now
tatiana
2018-07-10 14:46
Hm, when I click on ?Build? (for example https://jenkins.status.im/job/status-react/job/desktop/job/pull-requests/view/change-requests/job/PR-5072/) - I see message that build is sheduled, but cannot see it in Build queue
jakub
2018-07-10 14:46
let me see
jakub
2018-07-10 14:47
oh, master is also out
jakub
2018-07-10 14:47
nice
jakub
2018-07-10 14:47
okay, I'll finish some icinga configuration and I'll start work on properly deploying the master with more space, 40GB is ridiculous
tatiana
2018-07-10 14:49
Agree :slightly_smiling_face:
please ping me when I can trigger build again
jakub
2018-07-10 14:50
try now
tatiana
2018-07-10 14:51
it is building, thanks
jakub
2018-07-10 14:57
is the desktop stuff merged into `develop` or are you still using the `desktop` branch?
tatiana
2018-07-10 14:59
still using `desktop`, in process of migrating
jakub
2018-07-10 14:59
okay, I will target my fix at `desktop` then
jakub
2018-07-10 15:03
not perfect but will buy some time
tatiana
2018-07-10 15:07
awesome, thanks!
jakub
2018-07-10 15:07
hmmm, I need to test it with the manual build first, seems my syntax is off
tatiana
2018-07-10 15:08
OK, let me know when it?ll be ready
anna
2018-07-10 16:24
run nightly build on Jenkins twice and got error ``` ERROR: Error fetching remote repo 'origin'
hudson.plugins.git.GitException: Failed to fetch from https://github.com/status-im/status-react.git
at hudson.plugins.git.GitSCM.fetchFrom(GitSCM.java:888)
at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1155)
at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1186)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:113)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:85)
at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:75)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1$1.call(AbstractSynchronousNonBlockingStepExecution.java:47)
at hudson.security.ACL.impersonate(ACL.java:290)
at org.jenkinsci.plugins.workflow.steps.AbstractSynchronousNonBlockingStepExecution$1.run(AbstractSynchronousNonBlockingStepExecution.java:44)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: hudson.plugins.git.GitException: Command "git config remote.origin.url https://github.com/status-im/status-react.git" returned status code 255:
stdout:
stderr: error: could not lock config file .git/config: No space left on device ```
anna
2018-07-10 16:26
not sure if it's also because of no space on linux1 node cc @jakub if all builds will fail - should we inform in core channel?
jakub
2018-07-10 16:26
which build?
jakub
2018-07-10 16:27
oh lovely, now the macos one
anna
2018-07-10 16:28
is it because of desktop builds? I mean no free space
jakub
2018-07-10 16:28
I don't think you build desktop on macos
jakub
2018-07-10 16:28
or do you?
anna
2018-07-10 16:29
on macos node
anna
2018-07-10 16:29
macos1 to be precise
jakub
2018-07-10 16:29
right
jakub
2018-07-10 16:33
should be fine now
anna
2018-07-10 16:35
thank you Jacob :slightly_smiling_face: at least it can proceed to next stages now :slightly_smiling_face:
jakub
2018-07-10 16:36
actually, it doesnt seem like jenkins caused macos1 to run out of space
jakub
2018-07-10 16:36
not sure yet
anna
2018-07-10 17:03
https://jenkins.status.im/job/status-react/job/nightly/490/ nothing failed yet but `Deploy (Android)` takes more time then usual
jakub
2018-07-10 17:04
hm, true
jakub
2018-07-10 17:04
but that's the deploying stage
jakub
2018-07-10 17:04
maybe artifactory has issues
jakub
2018-07-10 17:04
seems to be up
jakub
2018-07-10 17:05
but I've seen this befroe
jakub
2018-07-10 17:05
no, it has plenty of space
jakub
2018-07-10 17:05
but this kind of hangup happened before
jakub
2018-07-10 17:05
and it's one of the reason why I was looking for a replacement for artifactory
jakub
2018-07-10 17:05
though I couldn't find anything sensible
jakub
2018-07-10 17:07
you might have to restart it
anna
2018-07-10 17:09
ok
anna
2018-07-10 17:11
stopped previous and will start new one soon. btw, is it normal that https://jenkins.status.im/job/status-react/job/desktop/job/manual/42/ takes more than hour?
jakub
2018-07-10 17:11
not rally
jakub
2018-07-10 17:11
really*
jakub
2018-07-10 17:11
maybe this si due to the space issues we had earlier
jakub
2018-07-10 17:11
killed it
jeluard
2018-07-16 06:26
Right looks like Jenkins had overdosis again.
anna
2018-07-16 07:51
@jakub good morning :slightly_smiling_face: any estimates on when Jenkins will be back? :slightly_smiling_face:
jakub
2018-07-16 07:51
oh, is it dead?
jakub
2018-07-16 07:51
damn
jakub
2018-07-16 07:53
I actually already have a replacement host for our current Jenkins Master, I just need to prepare a proper backup procedure so we don't lose anything when moving
jakub
2018-07-16 07:53
that might happen today or tomorrow
jakub
2018-07-16 07:53
then we'll have more disk space and proper monitoring
jakub
2018-07-16 08:23
oh, btw, forgot to say, jenkins is back, has been for a while :smile:
igor680
2018-07-16 09:21
@jakub I'd really appreciate any kind of `@here` message in #core for changes with Jenkins that might cause it planned outage :slightly_smiling_face:
jakub
2018-07-16 09:21
that wasn't a planned outage my dude
jakub
2018-07-16 09:21
I never said it was
jakub
2018-07-16 09:21
I just said I'm preparing a replacement server
igor680
2018-07-16 09:22
:slightly_smiling_face: yeah, but all the jobs will die, right? :slightly_smiling_face:
jakub
2018-07-16 09:22
when I replace the server? not really. I will probably do it when there are no jobs, probably outside of EU working hours
jakub
2018-07-16 09:23
not sure when exactly yet, right now I'm working with Pedro on mailserver canaries
igor680
2018-07-16 09:23
ah, then it is fine
still nice to know these kinds of plans just so they don't interfere with our release schedule
jakub
2018-07-16 09:24
sure, will keep that in mind
igor680
2018-07-16 09:24
so just in case if that doesn't happen today or tomorrow, let's move it to the next week...
jakub
2018-07-16 09:24
are we planning a realease sometime this week?
igor680
2018-07-16 09:25
on a normal cadence we release something each week, the release testing starts on Wed usually
jakub
2018-07-16 09:25
well, then I guess the best time for such a change would be the weekend =]
anna
2018-07-16 09:26
note that to start testing on Wednesday we need nightly build that is triggered outside of EU working hours
jakub
2018-07-16 09:27
right
anna
2018-07-16 09:27
it's also likely that we will need to trigger more builds during Wednesday working hours (at least it happened in the past releases)
jakub
2018-07-16 09:27
I just have to move the nightly to either a sooner or later hour
igor680
2018-07-16 09:27
yeah, for the releases the weekend is the best
igor680
2018-07-16 09:28
but it is negotiable of course, sometimes we can skip a release. just this week is not a good time for that :slightly_smiling_face:
jakub
2018-07-16 09:28
sure
andrey
2018-07-16 13:16
idk know who i should ask, @jakub @adamb @divan we have faucet service `http://51.15.45.169:3001/donate/` , i want to access it from `https://status-im.github.io/dapp/` how we can do that? now i have CORS error
jakub
2018-07-16 13:20
yeah, the service has to allow access from a different domain via a few HTTP CORS headers
jakub
2018-07-16 13:20
jakub
2018-07-16 13:22
it doesn't look like I have access to `51.15.45.169`
andrey
2018-07-16 13:35
:disappointed:
jakub
2018-07-16 13:35
maybe @adamb has access and can give it to me too?
andrey
2018-07-16 13:35
or @divan
adamb
2018-07-16 14:02
I have access
adamb
2018-07-16 14:03
@jakub I am gonna add your public key
jakub
2018-07-16 14:03
a friend with an ssh public key is a friend indeed
jakub
2018-07-16 14:03
cheers
jakub
2018-07-16 14:03
I can add the CORS setup to what's there already, but it's gonna be just a temporary patch, one of these days I'm gonna have to automate the faucet setup
adamb
2018-07-16 14:05
check out `root@51.15.45.169`
jakub
2018-07-16 14:05
bless your heart
jakub
2018-07-16 14:07
I see, so it was set up by this: https://github.com/status-im/status-cluster/blob/master/roles/faucet/templates/docker-compose.yml
jakub
2018-07-16 14:07
I will have to re-create the container under a different port and then put it behind a proxy to add the CORS header
jakub
2018-07-16 14:07
s*
jakub
2018-07-16 14:08
is there any danger in that?
jakub
2018-07-16 14:09
What domains do you want to allow CORS requests from? Just http://status-im.github.io ? or others?
jakub
2018-07-16 14:12
Also, are you in any way attached to port `3001` or can I just put the proxy at `80`/`443`?
jakub
2018-07-16 14:27
``` % curl -ski https://faucet.status.im/ -H 'Origin: https://status-im.github.io/' | grep -i 'access-control'
access-control-allow-origin: https://status-im.github.io/
access-control-allow-credentials: true
access-control-allow-methods: GET, POST, PUT, DELETE, OPTIONS
access-control-allow-headers: Accept,Authorization,Cache-Control,Content-Type,DNT,If-Modified-Since,Keep-Alive,Origin,User-Agent,X-Requested-With```
jakub
2018-07-16 14:27
there you go
jakub
2018-07-16 14:27
you might not see it yet because I created the dns record a short while ago
jakub
2018-07-16 14:28
also, right not it only allows `http://status-im.github.io`, what else should it?
andrey
2018-07-16 14:33
i think `http://status-im.github.io` is enough for now, thanks, i'm not sure about port
jakub
2018-07-16 14:33
well, i went with standard `443`, otherwise SSL would be an issue
jakub
2018-07-16 14:34
`65324 Ether` that's a lot of Ethere
andrey
2018-07-16 14:34
that's cool :slightly_smiling_face:
jakub
2018-07-16 14:35
sure
andrey
2018-07-16 14:36
cool thanks
jakub
2018-07-16 14:37
no problem my dude
lukasz949
2018-07-17 10:54
Hey, I can't find PR jenkins build (https://jenkins.status.im/job/status-react/job/pull%20requests/) for https://github.com/status-im/status-react/pull/5178
Any idea what has happened and how to trigger it? @jakub?
jakub
2018-07-17 10:58
there never was one
jakub
2018-07-17 11:15
now it happened
jakub
2018-07-17 11:15
I believe the issue was that there was a conflict
jakub
2018-07-17 11:16
Jenkins can't rebase a conflicting branch
lukasz949
2018-07-17 11:21
got it. Thanks!
rob018
2018-07-17 16:43
@jakub I've been trying to get the latest copy of master branch for the status open bounty app out into production, though the deploy logs point to the `openbounty-cluster` job rather than to `status-openbounty-app`. This may be by design? Was hoping you might be able to help when you have time. Thanks! https://jenkins.status.im/job/status-openbounty/job/status-openbounty-app/job/master/
jakub
2018-07-17 17:01
Not, sure, let me research this
rob018
2018-07-17 18:26
thanks!
jakub
2018-07-17 19:27
it seems like both are necessary
jakub
2018-07-17 19:27
the https://jenkins.status.im/job/status-openbounty/job/status-openbounty-app one pushes the docker image
jakub
2018-07-17 19:28
then the https://jenkins.status.im/job/status-openbounty/job/openbounty-cluster/ job runs the ansible role to update the docker container
jakub
2018-07-17 19:28
jakub
2018-07-17 19:29
so overcomplicated, the host could pull a new image when available with watchtower
rob018
2018-07-17 20:13
thanks for clarifying. in your opinion it does look like the cluster is properly pulling the latest built docker image for the master branch? guess I can try running the image here: https://hub.docker.com/r/statusim/openbounty-app/ to verify if the expected changes are there?
jakub
2018-07-17 20:13
you could
jakub
2018-07-17 20:13
but let me check the host
rob018
2018-07-17 20:14
thank you! sorry for all this :grimacing:
jakub
2018-07-17 20:14
it's what I'm here for my dude
rob018
2018-07-17 20:14
:raised_hands:
jakub
2018-07-17 20:14
```66887d244a76 statusim/openbounty-app:master "/usr/bin/java -Du..." 4 hours ago Up 4 hours testing
```
jakub
2018-07-17 20:14
"Up 4 hours"
jakub
2018-07-17 20:14
```Success Build #272 (Jul 17, 2018 3:58:53 PM)```
jakub
2018-07-17 20:15
```Started 4 hr 15 min ago```
rob018
2018-07-17 20:15
ok, that does sound right actually.
jakub
2018-07-17 20:15
sounds right
rob018
2018-07-17 20:15
thanks for verifying!
jakub
2018-07-17 20:15
cheers
jakub
2018-07-17 20:16
I'm gonna add it to my todo to simplify that CI setup
jakub
2018-07-17 20:16
the second job is really not necessary
rob018
2018-07-17 20:17
that would be great. there's always been some ambiguity on my end on how it got deployed so will look into this too now that you've shed more light on it. much appreciated
jakub
2018-07-17 20:18
I have to setup a new consul cluster in hong kong tomorrow, finish oauth2 changes, and prepare for jenkins migration, but I should get to it on Thursday
jakub
2018-07-17 20:18
it shouldn't be much work
rob018
2018-07-17 20:18
:sunglasses: good stuff on all accounts!
rob018
2018-07-17 20:19
are you migrating things to or away from jenkins?
jakub
2018-07-17 20:19
I'm moving our jenkins setup to a host with proper monitoring and more disk space
jakub
2018-07-17 20:19
we've been having to many issues with disk space lately
rob018
2018-07-17 20:19
desktop, lol :eyes:
jakub
2018-07-17 20:19
moving away from jenkins would be a much, much bigger undertaking =]
jakub
2018-07-17 20:20
yeah, well, a jenkins master with 30 gb of space is just silly
rob018
2018-07-17 20:20
hah, yea was gonna say
jakub
2018-07-17 20:20
https://www.youtube.com/watch?v=TkCZ1wCi6As - here, grab a lovely City Pop compilation
jakub
2018-07-17 20:21
so good
rob018
2018-07-17 20:22
was hooked before I finished reading "a bright future, naive optimism, and glamorous lifestyles that would never last"
rob018
2018-07-17 20:22
:joy:
jakub
2018-07-17 20:22
it's a pretty good description
rob018
2018-07-17 20:23
a description that applies to a surprisingly large amount of things...
jakub
2018-07-17 20:24
yeah, especially "can evoke ghosts of lives never lived"
rob018
2018-07-17 22:24
so what is interesting is that the only host in our `status-cluster` repo for open-bounty is `http://testing.openbounty.status.im` and it looks like the task that is configured w/ the docker image is working properly there (https://testing.openbounty.status.im/ has the updated code!) It does leave me in the dark though about how updates actually make it to prod, as I can't find the host https://openbounty.status.im/ in `status-cluster`. may have to put on the City Pop Album again lol
jakub
2018-07-18 08:20
oh, so the host the job pushes to is a test host, I wasn't aware of that
jakub
2018-07-18 08:20
right
```
http://testing.openbounty.status.im. 300 IN A 51.15.45.92
http://openbounty.status.im. 300 IN A 139.162.11.12
```
jakub
2018-07-18 08:21
I guess updates to prod are done by hand
jakub
2018-07-18 08:21
once we are sure things are working in testing
jakub
2018-07-18 08:46
If you are fine with the changes in http://testing.openbounty.status.im I can push them toprod
jeluard
2018-07-18 14:04
Hey @jakub ! I'm wondering how often the github polls is triggered here: https://jenkins.status.im/job/misc/job/docs.status.im/GitHubPollLog/
jakub
2018-07-18 14:04
github talks to jenkins, not the other way around
jakub
2018-07-18 14:05
github has configured a webhook that it posts to on various events in the repo
rob018
2018-07-18 14:10
thanks jacob! yes I think everything on http://testing.openbounty.status.im looks good, so am fine with that heading to prod itself
jakub
2018-07-18 14:10
okay, give me a few min, I want to finish something
rob018
2018-07-18 14:10
lmk if you need hand or if its easier to offload that process to me also :thumbsup:
rob018
2018-07-18 14:10
gotcha. sounds great
jakub
2018-07-18 14:11
I will have to change how it's deployed to let you do that, no time for now, I'll do it by hand for now
rob018
2018-07-18 14:16
cool. thanks jacob!
jeluard
2018-07-18 14:39
Ok well doesn't work then ..
jeluard
2018-07-18 14:40
My latest commit has not triggered a new build
jeluard
2018-07-18 15:22
ping @jakub :sweat_smile:
jakub
2018-07-18 15:22
where? you mean in a pull request?
jakub
2018-07-18 15:22
which one?
jeluard
2018-07-18 15:24
I pushed some commits but it's not been published
jakub
2018-07-18 15:25
there was a webhook push at `?? 2018-07-18 15:44:2`
jakub
2018-07-18 15:26
hmmm
jakub
2018-07-18 15:26
```Jul 18, 2018 1:44:26 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/status-im/docs.status.im from 192.30.252.36 ? https://jenkins.status.im:8080/github-webhook/
Jul 18, 2018 1:44:26 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run
Poked misc � http://docs.status.im```
jakub
2018-07-18 15:26
`Jul 18, 2018 1:44:26`
jakub
2018-07-18 15:26
that's pretty old
jakub
2018-07-18 15:27
```Started on Jul 18, 2018 1:44:26 PM
Started by event from 192.30.252.36 ? https://jenkins.status.im:8080/github-webhook/ on Wed Jul 18 13:44:26 UTC 2018
Using strategy: Default
[poll] Last Built Revision: Revision 9047a234348bb0fe2e0726acec2edbb0dc275fb2 (refs/remotes/origin/fix/upgrade-docosaurus)```
jakub
2018-07-18 15:27
right
jakub
2018-07-18 15:27
oh
jakub
2018-07-18 15:28
it's the branch
jakub
2018-07-18 15:28
that's my fault I think, I used it to test `fix/upgrade-docosaurus`
jakub
2018-07-18 15:28
try now
jakub
2018-07-18 15:28
push anything
jakub
2018-07-18 15:28
to `develop`
jeluard
2018-07-18 15:30
:+1:
jakub
2018-07-18 15:30
cool, sorry for that =]
jeluard
2018-07-18 15:45
Hmm still not happy?
jakub
2018-07-18 15:45
`No changes`
jakub
2018-07-18 15:45
wat
jakub
2018-07-18 15:45
`9047a234348bb0fe2e0726acec2edbb0dc275fb2` is this the right commit?
jeluard
2018-07-18 15:47
Doesn't look so
jakub
2018-07-18 15:48
the webhook request looks right
jakub
2018-07-18 15:48
```{
"ref": "refs/heads/develop",
"before": "c4dbcb67a28a5bbc04c368be5c25cd229c908de5",
"after": "d5e9833eb66325e16b5ff8468a6b49d15b64bf9b",
```
jakub
2018-07-18 15:49
```Jul 18, 2018 3:42:31 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber onEvent
Received PushEvent for https://github.com/status-im/docs.status.im from 192.30.252.39 ? https://jenkins.status.im:8080/github-webhook/
Jul 18, 2018 3:42:31 PM INFO org.jenkinsci.plugins.github.webhook.subscriber.DefaultPushGHEventSubscriber$1 run
Poked misc � http://docs.status.im```
jakub
2018-07-18 15:49
wat
jakub
2018-07-18 15:49
I changed the branch but it still mentions `refs/remotes/origin/fix/upgrade-docosaurus`
jakub
2018-07-18 15:49
why...
jeluard
2018-07-18 15:50
Did you plugged it off and on again?
jakub
2018-07-18 15:50
ran it by hand and it worked
jakub
2018-07-18 15:50
I guess it somehow cached the branch name and I had to run it to forget the about other one
jakub
2018-07-18 15:50
freakin' voodoo
jeluard
2018-07-18 15:51
:fingers-crossed:
jakub
2018-07-18 15:56
Jesus, this looks far more complicated that it needs to be
jakub
2018-07-18 15:56
looks like prod doesn't run from a docker container
jakub
2018-07-18 15:57
it's some kind of Frankenstein's monster setup done by hand that I'm wary of touching
jakub
2018-07-18 15:57
I REALLY need to do a proper setup for openbounty using docker :confused:
rob018
2018-07-18 18:22
k, haha agreed on the current state of affairs. lmk if I can help. even moreso than deployment is my fear that those prod configs aren't backed up anywhere too. seems like only a matter of time before something goes south :grimacing: thanks for checking it out initially! keep me posted if I can help in any transition to a saner process
jakub
2018-07-18 18:23
I want to do it tomorrow. I hope I can get the consul cluster going in Hong Kong today, though I will have to make an Ubuntu 18.04 image for Alibaba Cloud with Packer because they don't support it out of the box
jakub
2018-07-18 18:24
freakin' yak shaving
rob018
2018-07-18 18:27
all good man! thanks for getting it on the list. (and interesting. wonder what everyone runs on alibaba instead :thinking_face: )
jakub
2018-07-18 18:27
16.04 is the default
jakub
2018-07-18 18:28
they are also behind on images for Fedora and CentOS
jakub
2018-07-18 18:28
and unlike AWS they have not "community" section for images
rob018
2018-07-18 18:29
:grimacing: do they have a saner pricing model? is that the draw for using them?
jakub
2018-07-18 18:30
I think the draw is that they have servers in China that don't get firewalled all the freakin' time
jakub
2018-07-18 18:30
when I worked in AWS we would hear about another big IP section of a DC being blocked by chinese every week or two
rob018
2018-07-18 18:32
it all makes sense now! cool stuff, hadn't been anywhere that was setting up on alibaba cloud before :flag-cn:
sergii
2018-07-20 11:06
Hey @jakub could you help us to understand why nightly build fails? https://jenkins.status.im/job/status-react/job/nightly/505 (on the ` bundle exec fastlane ios nightly`) step
sergii
2018-07-20 11:07
```[11:58:23]: Distributing new build to testers: 0.9.23 - 6224
+---------------+-------------+
| [33mLane Context[0m |
+---------------+-------------+
| PLATFORM_NAME | ios |
| LANE_NAME | ios nightly |
+---------------+-------------+
[11:58:24]: [31mServer error got 500```
sergii
2018-07-20 11:08
`[32mPlease update using `bundle update fastlane`
jakub
2018-07-20 12:41
im back, let me see
jakub
2018-07-20 12:41
```[!] The request could not be completed because:
Server error got 500```
jakub
2018-07-20 12:44
```[11:58:24]: [31mServer error got 500[0m
+------+----------------------+-------------+
| [32mfastlane summary[0m |
+------+----------------------+-------------+
| Step | Action | Time (in s) |
+------+----------------------+-------------+
| ? | [31mupload_to_testflight[0m | 234 |
+------+----------------------+-------------+
```
jakub
2018-07-20 12:44
I do want to upgrade fastlane, but I've been asked to hold off on that until we do the current release
jakub
2018-07-20 12:45
it seems like what failed here is the `upload_to_testflight` step
jakub
2018-07-20 12:52
I see, it's been like that since #501
jakub
2018-07-20 12:52
`Jul 19, 2018 2:59 AM`
jakub
2018-07-20 12:53
Yeah, @igor680 asked me to update it on Monday evening
jakub
2018-07-20 12:53
to avoid messing with release builds
igor680
2018-07-20 12:53
Yep, Monday
jakub
2018-07-20 12:53
So I guess nightly is broken for now
igor680
2018-07-20 12:54
It still uploads it just fine, just fails at the last check
jakub
2018-07-20 12:54
I'm not even sure where the `500` comes from
jakub
2018-07-20 12:54
it just says there's a `500`, but from where? no idea
jakub
2018-07-20 12:54
right
jakub
2018-07-20 12:54
I can see the iTunes upload finishes fine
jakub
2018-07-20 12:54
just before the error
jakub
2018-07-20 12:54
```[11:57:52]: Waiting for iTunes Connect to finish processing the new build (0.9.23 - 6224)
[11:58:23]: [32mSuccessfully finished processing the build 0.9.23 - 6224[0m```
jakub
2018-07-20 12:55
Also, we really need to give fastlane some kind of flag to stop using colors :confused:
jakub
2018-07-20 12:55
```export FASTLANE_DISABLE_COLORS=1
```
igor680
2018-07-20 13:08
Yep, I successfully submitted it to review so it works
jakub
2018-07-20 13:08
cool
sergii
2018-07-20 13:59
Thanks guys. So, next Monday (23rd July) we expect it to get back to work, right?
jakub
2018-07-20 14:00
yup
nastya
2018-07-23 06:26
@jakub any magic should be done to bring Jenkins to life? https://jenkins.status.im/job/status-react/job/nightly/509/ fails
jakub
2018-07-23 06:29
I will try updating fastlane later today, if you look at the logs the artifacts actually get uploaded fine
nastya
2018-07-23 06:29
:eyes:
jakub
2018-07-23 06:30
It fsils at the very end, after the build and upload finishes
nastya
2018-07-23 06:30
ah i see the artifacts yeah. was confused with the fails though
jakub
2018-07-23 06:31
Yeah, it just says it gets 500, but why?
nastya
2018-07-23 06:31
�\_(?)_/�
nastya
2018-07-23 07:08
@jakub no iOS artifact created for some reason
jakub
2018-07-23 07:08
how can that be true if I can see in the logs ```[07:24:27]: Waiting for iTunes Connect to finish processing the new build (0.9.23 - 6236)
[07:24:57]: [32mSuccessfully finished processing the build 0.9.23 - 6236[0m```
nastya
2018-07-23 07:09
no diawi link (
jakub
2018-07-23 07:09
ah, well, sorry, you will have to wait till afternoon
jakub
2018-07-23 07:09
then I'll work on fastlane
nastya
2018-07-23 07:09
okay ! thanks!
tatiana
2018-07-23 07:14
@jakub also at attempt to download build from https://jenkins.status.im/job/status-react/job/desktop/job/pull-requests/view/change-requests/job/PR-5239/ - getting error ?Incomplete file?,if I?m trying to trigger new build - nothing happens.
Should I wait until tomorrow as well or there is way to get artifacts faster?
jakub
2018-07-23 07:15
when are you getting the error? when installing the app?
jakub
2018-07-23 07:15
ha
jakub
2018-07-23 07:15
of course it's space
jakub
2018-07-23 07:15
```/dev/vda1 39G 39G 0 100% /
```
jakub
2018-07-23 07:18
i've restarted it
jakub
2018-07-23 07:18
I really want to replace that host :confused:
tatiana
2018-07-23 07:18
yep, agree) like we are running in circle every week
jakub
2018-07-23 07:18
but I'm flying to the US on wednestday
jakub
2018-07-23 07:18
and I'm on holidays until Monday
jakub
2018-07-23 07:19
I kinda don't want to do it when I'm leaving for a few days
jakub
2018-07-23 07:44
can you check now?
igor680
2018-07-23 08:20
@jakub @adamb I guess we have some unclean state for our go repo. I can't build status-go-manual job. Can't update status-go w/o it.
```
grouped write of manifest, lock and vendor: scratch directory /home/jenkins/workspace/status-go/status-go-manual/src/github.com/status-im/status-go/.vendor-new already exists, please remove it
Makefile:242: recipe for target 'dep-ensure' failed
make: *** [dep-ensure] Error 1
```
jakub
2018-07-23 08:21
let me see this
jakub
2018-07-23 08:21
which slave is that?
igor680
2018-07-23 08:21
`Running on linux2 in /home/jenkins/workspace/status-go/status-go-manual`
jakub
2018-07-23 08:22
thanks
jakub
2018-07-23 08:24
wat
jakub
2018-07-23 08:24
```grouped write of manifest, lock and vendor: failed to export http://github.com/ethereum/go-ethereum: fatal: failed to unpack tree object dea1ce052a10cd7d401a5c04f83f371a06fe293c
```
igor680
2018-07-23 08:25
�\_(?)_/�
igor680
2018-07-23 08:25
I don't have this issue locally
igor680
2018-07-23 08:25
just checked
jakub
2018-07-23 08:25
sounds like a corrupt repo
igor680
2018-07-23 08:26
maybe kill it and do a fresh checkout?
jakub
2018-07-23 08:26
the commit exists
jakub
2018-07-23 08:26
worth a try
jakub
2018-07-23 08:30
same thing again
jakub
2018-07-23 08:30
`failed to unpack tree object dea1ce052a10cd7d401a5c04f83f371a06fe293c
: exit status 128`
jakub
2018-07-23 08:33
hmmm, I can checkout the commit just fine by hand
igor680
2018-07-23 08:33
and it works on my local repo :disappointed:
igor680
2018-07-23 08:35
it might be some lock files left over
jakub
2018-07-23 08:35
not really
jakub
2018-07-23 08:35
I deleted the whole dir
jakub
2018-07-23 08:36
changed on Friday
igor680
2018-07-23 08:38
hmm, *Jun* 20
jakub
2018-07-23 08:39
oh, my bad :smile:
jakub
2018-07-23 08:40
wtf
jakub
2018-07-23 08:41
`.vendor-new/github.com/ethereum/go-ethereum` is empty
jakub
2018-07-23 08:45
Why is this saying `1.8.5`:
jakub
2018-07-23 08:45
and Gopkg.lock says `1.8.11`:
igor680
2018-07-23 08:46
no clue.
igor680
2018-07-23 08:46
@adamb @divan?
igor680
2018-07-23 08:46
mb we need to update `dep` to the latest version on the builder?
jakub
2018-07-23 08:47
`dep`?
jakub
2018-07-23 08:48
a
jakub
2018-07-23 08:48
`Dep is a tool for managing dependencies for Go projects`
igor680
2018-07-23 08:48
yeh
igor680
2018-07-23 08:49
@jakub we probably need `v0.4.1`
jakub
2018-07-23 08:49
```$ dep version
dep:
version : devel
build date :
git hash :
go version : go1.9.4
go compiler : gc
platform : linux/amd64
features : ImportDuringSolve=false```
igor680
2018-07-23 08:50
but I'd rather installed some release version
jakub
2018-07-23 08:50
right
divan
2018-07-23 08:55
`make dep-install`
divan
2018-07-23 08:55
we have this in our Makefile
igor680
2018-07-23 08:56
It just doesn't feel right that we just use the latest master of dep on a CI server...
divan
2018-07-23 08:56
@igor680 you mean we should not use latest dep? I'm not sure I got what's the problem.
jakub
2018-07-23 08:57
```go get -u http://github.com/golang/dep/cmd/dep``` this is what `make dep-install` does
divan
2018-07-23 08:57
Exactly
jakub
2018-07-23 08:57
I'm not evne sure how I can enforce a specific version
igor680
2018-07-23 08:57
@divan that we shouldn't use the latest commit of dep, and instead use a latest _release_ :slightly_smiling_face:
divan
2018-07-23 08:57
Oh. Is it a bug in latest dep or what?
jakub
2018-07-23 08:58
not sure
jakub
2018-07-23 08:58
but it doesn't seem to change anything
jakub
2018-07-23 08:58
still shows version as `devel`
igor680
2018-07-23 08:58
yeah, using dep to update dep :slightly_smiling_face:
jakub
2018-07-23 08:58
then how do I do it with `go get`?
igor680
2018-07-23 08:58
you need to remove `dep` from `$GOPATH/bin`
jakub
2018-07-23 08:58
what's wrong with that? `npm` can update `npm`
jakub
2018-07-23 08:58
`apt` can update `apt`
jakub
2018-07-23 08:59
`yum` cam update `yum`
jakub
2018-07-23 08:59
that's normal
jakub
2018-07-23 08:59
I see
divan
2018-07-23 08:59
@jakub what are you trying to do?
jakub
2018-07-23 08:59
enforce `0.4.1`?
igor680
2018-07-23 08:59
yeah, `go get` only updates to the `HEAD` of the master branch :slightly_smiling_face:
divan
2018-07-23 08:59
dep is not a package manager, is a vendoring tool
jakub
2018-07-23 08:59
I see
igor680
2018-07-23 09:00
but
> On other platforms you can use the install.sh script:
`$ curl https://raw.githubusercontent.com/golang/dep/master/install.sh | sh`
igor680
2018-07-23 09:00
@divan
> It is strongly recommended that you use a released version. Release binaries are available on the releases page.
jakub
2018-07-23 09:01
okay, that worked
jakub
2018-07-23 09:01
```$ /home/jenkins/workspace/status-go/status-go-manual/bin/dep version
dep:
version : v0.4.1
build date : 2018-01-24
git hash : 37d9ea0a
go version : go1.9.1
go compiler : gc
platform : linux/amd64```
igor680
2018-07-23 09:01
yay, awesome
jakub
2018-07-23 09:01
well, at least now it fails on some code errors
jakub
2018-07-23 09:05
seems to be working
divan
2018-07-23 09:05
sounds like `dep-install` should be changed to use releases
jakub
2018-07-23 09:05
yeah
jakub
2018-07-23 09:08
brb, out to postoffice
jakub
2018-07-23 09:14
Or not brb, because of coyrse theres a queue of old people who will never learn how to use internet banking
nastya
2018-07-23 13:25
@jakub any luck with nightly?
jakub
2018-07-23 14:20
oh yeah, let me do something for Ivan and then I'll get to it
jeluard
2018-07-23 14:43
@jakub Can you help with moving http://docs-staging.status.im to http://docs.status.im ?
jakub
2018-07-23 15:06
in a few minuts
jeluard
2018-07-23 15:07
Sure no rush!
jakub
2018-07-23 15:19
done
jeluard
2018-07-23 15:36
Awesome thks!
jeluard
2018-07-23 15:37
Just add a beer to my note of beers I owe you ! :beers:
jakub
2018-07-23 15:38
I should start a spreadsheet for those
jakub
2018-07-23 15:39
Looks like upgrade to `2.100.1` worked:
https://jenkins.status.im/job/status-react/job/nightly/510/console
nastya
2018-07-23 15:44
:+1:
2018-07-24 07:56
@andreap commented on @nastya?s file https://status-im.slack.com/files/U5EUG5YTG/FBV6CMA0Z/nightly_could_not_be_built_again.java: seems like `prepare-for-mobile.sh` is not linking things properly (or similar), I don't have access to jenkins so a bit difficult to debug from here
2018-07-24 07:58
@nastya commented on @nastya?s file https://status-im.slack.com/files/U5EUG5YTG/FBV6CMA0Z/nightly_could_not_be_built_again.java: @yenda has already found the root cause i believe
nastya
2018-07-24 11:40
nightly build still fails https://jenkins.status.im/job/status-react/job/nightly/514/ @jakub @yenda
yenda
2018-07-24 11:41
@jakub something to do with credentials ?
jakub
2018-07-24 11:43
```[12:21:57]: Server error got 500```
jakub
2018-07-24 11:43
ehhh....
jakub
2018-07-24 11:43
that's all we have to go by
jakub
2018-07-24 11:43
```fastlane 2.100.1 is available. You are on 2.88.0.```
jakub
2018-07-24 11:43
wat
jakub
2018-07-24 11:43
I upgraded it
jakub
2018-07-24 11:44
```administrators-Mac-Pro:~ jenkins$ fastlane --version
fastlane installation at path:
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.100.1/bin/fastlane
-----------------------------
[?] ?
fastlane 2.100.1
```
jakub
2018-07-24 11:44
what fresh madness is this
jakub
2018-07-24 11:47
ooooh
jakub
2018-07-24 11:47
```administrators-Mac-Pro:nightly jenkins$ bundle exec fastlane --version
fastlane installation at path:
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.88.0/bin/fastlane
-----------------------------
[?] ?
fastlane 2.88.0
```
jakub
2018-07-24 11:48
ooooooooh
nastya
2018-07-24 11:48
:okay:
jakub
2018-07-24 11:48
let me change that
nastya
2018-07-24 11:49
:pray:
igor680
2018-07-24 13:12
did it fix the issue?
jakub
2018-07-24 13:13
I'm running a nightly with a branch right now
jakub
2018-07-24 13:13
oh wait, it failed
jakub
2018-07-24 13:15
we'll see if it works
igor680
2018-07-24 13:41
still `[14:39:14]: Server error got 500` :disappointed:
jakub
2018-07-24 13:41
oh for fucks sake
jakub
2018-07-24 13:43
I'm gonna try adding `--debug` to it
jakub
2018-07-24 13:44
or `--verbose`
igor680
2018-07-24 13:44
yeah
jakub
2018-07-24 15:04
Well that's not very helpful:
```
INFO [2018-07-24 15:59:27.10]: Waiting for App Store Connect to finish processing the new build (0.9.23 - 6245)
INFO [2018-07-24 16:00:00.32]: Build doesn't show up in the build list anymore, waiting for it to appear again (check your email for processing issues if this continues)
INFO [2018-07-24 16:00:34.18]: Successfully finished processing the build 0.9.23 - 6245
INFO [2018-07-24 16:00:34.18]: Distributing new build to testers: 0.9.23 - 6245
WARN [2018-07-24 16:00:35.03]: Lane Context:
INFO [2018-07-24 16:00:35.03]: {:PLATFORM_NAME=>:ios, :LANE_NAME=>"ios nightly"}
ERROR [2018-07-24 16:00:35.03]: Server error got 500
INFO [2018-07-24 16:00:35.03]: Successfully generated documentation at path '/Users/jenkins/workspace/status-react/nightly/fastlane/README.md'
```
jakub
2018-07-24 15:05
```bundler: failed to load command: fastlane (/usr/local/bin/fastlane)
Spaceship::InternalServerError: [!] The request could not be completed because:
Server error got 500
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.100.1/spaceship/lib/spaceship/test_flight/client.rb:323:in `handle_response'
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.100.1/spaceship/lib/spaceship/test_flight/client.rb:50:in `get_build'
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.100.1/spaceship/lib/spaceship/test_flight/build.rb:103:in `find'
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.100.1/spaceship/lib/spaceship/test_flight/build.rb:139:in `reload'
/usr/local/lib/ruby/gems/2.5.0/gems/fastlane-2.100.1/spaceship/lib/spaceship/test_flight/build.rb:187:in `export_compliance'
...```
jakub
2018-07-24 15:06
who has access to the App Store Testflight console?
jakub
2018-07-24 15:06
maybe we can find something about this `500` error there
jakub
2018-07-24 17:53
I'm flying out to New York tomorrow and I won't be available till Monday, so someone else will have to pick this up
jakub
2018-07-24 17:54
I believe following up with App Store Testflight to see if you can find any error explanation there
jakub
2018-07-24 17:54
because based off of what `fastlane` says I have no idea whatsoever what this is
jakub
2018-07-24 17:54
my assumption is that something is not right with the app we are uploading and the system rejects it with a 500
anna
2018-07-25 13:02
because @jakub is away we need someone else to fix nightly build @oskarth @igor680 https://status-im.slack.com/archives/C8M1Z25J8/p1532454839000011?thread_ts=1532432452.000270&cid=C8M1Z25J8
anna
2018-07-25 15:34
no issues atm, next nightly build is ok https://jenkins.status.im/job/status-react/job/nightly/521/
adamb
2018-07-26 09:31
/kudos @roman for configuring macOS machines in Jenkins! :clap:
igor680
2018-07-26 09:34
awesome! so both macos 2 and macos 3 are up and running now?
roman
2018-07-26 10:05
not yet
roman
2018-07-26 10:05
only macos2
roman
2018-07-26 10:05
i?m working on macos3 and docs
roman
2018-07-26 13:37
ok, macos3 works now too
vitaliy
2018-07-27 14:27
@maxr @volodymyr.kozieiev @rob018 @tatiana @oskarth can we have a discussion on the issues currently affecting Jenkins builds for desktop project? There is some manual effort needed now + large size of desktop builds frequently causes problems.
Ideally we?d want to build mobile/desktop for all PRs in `status-react` repo, right now this is not possible
maxr
2018-07-27 14:32
btw Jenkins Pipeline for desktop PRs is turned off at the moment because of above issues and we already starting receiving issues
jakub
2018-07-30 14:19
I will be replacing Jenkins master sometime this week, o can do the slaves to, but I need to know what steps I have to take to prepare the Linux slaves for desktop builds
jakub
2018-07-30 14:20
Do we have some doc on what needs to be installed for desktop builds to work?
maxr
2018-07-30 14:42
Hey @jakub will it be possible somehow to clone current working setup of linux1 slave? I've started creation of setup doc ( https://github.com/status-im/react-native-desktop/wiki/Jenkins-env-setup-to-build-react-native-desktop-based-app ) but it's not up-to-date and doesn't describe setup for Linux ... if not, I will try to gather the info what I've did for linux1
jakub
2018-07-30 14:44
please do
jakub
2018-07-30 14:45
I want to be able to set it up in Ansible but I need to know all the necessary steps
jakub
2018-07-30 14:45
it doesn't have to be perfect, if we find anything missing I'll just be fixing as we go
jakub
2018-07-30 14:46
for example, do we need some very specific QT dev headers verisons?
maxr
2018-07-30 14:51
Yes, Qt should be installed from official repo ( apt doesn't work on Linux and brew doesn't work on MacOS ). non-official builds have issue with updating binaries to have correct paths to point to bundled libs
But official repo provides only GUI based installers
Because of that I've just manually copied all Qt files to the slaves machines
jakub
2018-07-30 14:51
waaaaaaat
jakub
2018-07-30 14:52
that sounds awful, there must be a better way
maxr
2018-07-30 14:55
it should be :slightly_smiling_face:
jakub
2018-07-30 14:55
so what files specifically do you need?
jakub
2018-07-30 14:55
where did you put them?
maxr
2018-07-30 14:55
I will create some kind of doc for that and let you know
jakub
2018-07-30 14:56
awesome, no rush though, master comes first
maxr
2018-07-30 14:56
:+1: thanks!
nastya
2018-08-01 08:30
can't publish links : )
chad
2018-08-01 08:35
Is jenkins having issues? Lots of builds failing
roman
2018-08-01 08:39
ok, so issues are probably related to keychain unlocking
roman
2018-08-01 08:39
macos2 should be working fine
roman
2018-08-01 08:40
and i have no idea why macos3 doesn?t work properly, ipa form that machine crash on start. It?s not related to signing thing which i was working on previous week..
roman
2018-08-01 08:41
so what happened to macos2 is related to https://status-im.slack.com/archives/CAM2RMY65/p1532707668000329
roman
2018-08-01 08:42
as we had to reconnect slaves yesterday, we had to unlock keychain again
chad
2018-08-01 08:42
@roman I have this PR. How can I run it on Macos2?
roman
2018-08-01 08:43
why do you need to run it on macos2?
chad
2018-08-01 08:44
Hmm perhaps I misunderstood. I thought you meant there was a way to switch to a different jenkins host and successfully build
roman
2018-08-01 08:44
anyway, if you want to specially run it on macos2 you can replace `macos_pr` with `macos2` here https://github.com/status-im/status-react/blob/develop/Jenkinsfile#L20
roman
2018-08-01 08:44
no need to switch
roman
2018-08-01 08:45
the idea is that we have `macos_pr` label which is assigned to macos1 and macos2 at the moment
roman
2018-08-01 08:45
and PR will be assembled there
roman
2018-08-01 08:45
unfortunately macos3 doesn?t work properly atm, so i removed macos_pr label form this machine
chad
2018-08-01 08:58
So should it work if I just rerun on jenkins?
chad
2018-08-01 09:29
I re-ran and got a different error
```FAILURE: Build failed with an exception.
* What went wrong:
A problem occurred configuring project ':app'.
> The SDK directory '/usr/local/share/android-sdk' does not exist.
```
tatiana
2018-08-06 07:03
I cannot download build from https://ci.status.im/job/status-react/job/desktop/job/pull-requests/view/change-requests/job/PR-5363/.
On the middle I've got network error, @volodymyr.kozieiev got same error. Can we fix it somehow? It blocks testing because I couldn?t build `.dmg` build at all for desktop.
cc @maxr
maxr
2018-08-06 07:18
@tatiana I've just was able to download the .dmg file without any error from Windows ... will try to download and run on Mac in a moment
tatiana
2018-08-06 07:19
hm? thanks! keep me updated
maxr
2018-08-06 07:22
from PR you have mentioned, .dmg file https://ci.status.im/job/status-react/job/desktop/job/pull-requests/view/change-requests/job/PR-5363/lastSuccessfulBuild/artifact/StatusImPackage/StatusIm.dmg downloads fine for me at the moment ( last time when you mentioned the error, I also got it ) ... and app starts good as well
tatiana
2018-08-06 07:25
for me it is not working?
tatiana
2018-08-06 07:37
OK, this one can be downloaded.
https://ci.status.im/job/status-react/job/desktop/job/pull-requests/view/change-requests/job/PR-5330/
tatiana
2018-08-06 07:39
@maxr what?s next? AFAIK without this PR we cannot build Mac OSx at all.
maxr
2018-08-06 07:42
@tatiana do you have ability to try to download required .dmg from mobile, for example?
tatiana
2018-08-06 07:42
I couldn?t from safari, but I can try from IOS.
maxr
2018-08-06 07:46
I meant, if to try another device and another Internet provider
tatiana
2018-08-06 07:58
:tada: can directly transfer it using AirDrop. Success!
tatiana
2018-08-06 08:03
@maxr can you please merge it https://github.com/status-im/status-react/pull/5363?
maxr
2018-08-06 08:17
@tatiana can we wait also for @jakub input on that ... he is upgrading Jenkinsfile script heavily to work with new Jenkins setup
tatiana
2018-08-06 08:36
OK, let?s wait
tatiana
2018-08-06 09:00
last nightly was failed, seems again keychain problem:
```
[nightly] Running shell script
+ security unlock-keychain -p **** login.keychain
[Pipeline] sh
[nightly] Running shell script
+ xcodebuild -exportArchive -exportPath status_appstore -archivePath status.xcarchive -exportOptionsPlist /Users/jenkins/archive-release.plist
error: Couldn't load -exportOptionsPlist: The file "archive-release.plist" couldn't be opened because there is no such file.
Error Domain=NSCocoaErrorDomain Code=260 "The file "archive-release.plist" couldn't be opened because there is no such file." UserInfo={NSFilePath=/Users/jenkins/archive-release.plist, NSUnderlyingError=0x7f8812a792a0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
** EXPORT FAILED **
[Pipeline] }
[Pipeline] // withCredentials
[Pipeline] }
[Pipeline] // stage
[Pipeline] slackSend
run slackstepsend, step null:false, desc BZU9o3LpIcV1ST0RFDhXjh7L:true
Slack Send Pipeline step configured values from global config - baseUrl: true, teamDomain: true, token: true, channel: true, color: false
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timeout
[Pipeline] End of Pipeline
ERROR: script returned exit code 70
Finished: FAILURE
```
What can we do now to trigger build successfully (tried to rerun, got same error)?
nastya
2018-08-06 09:49
if @jakub is out who can assist with nightly? @yenda @janherich can u suggest someone?
jacqueswww
2018-08-06 10:19
@jakub quick questions I had while looking at grafana the other day:
1.) Looking at host average, why does ams node-07 have higher cpu usage than the rest?
2.) Looking at the statusd metrics, P2P inbound and outbound traffic - is it in bits or bytes per second?
graeme
2018-08-06 10:32
has joined #devops
jakub
2018-08-06 11:47
I'm not out, I'm in us timezone
jakub
2018-08-06 11:47
BRB, shower
nastya
2018-08-06 11:48
eh i misread your status then :slightly_smiling_face: :rubber-duck:
jakub
2018-08-06 11:48
I'll get back to you on that
jakub
2018-08-06 12:10
Hmmm, I could download both just fine
jakub
2018-08-06 12:11
what was the error you got?
jakub
2018-08-06 12:11
I'm in the dev meeting but nothing is happening
jakub
2018-08-06 12:11
please tell me whats up with nightly
jakub
2018-08-06 12:12
the latest one?
jakub
2018-08-06 12:12
```+ xcodebuild -exportArchive -exportPath status_appstore -archivePath status.xcarchive -exportOptionsPlist /Users/jenkins/archive-release.plist
error: Couldn't load -exportOptionsPlist: The file ?archive-release.plist? couldn?t be opened because there is no such file.
Error Domain=NSCocoaErrorDomain Code=260 "The file ?archive-release.plist? couldn?t be opened because there is no such file." UserInfo={NSFilePath=/Users/jenkins/archive-release.plist, NSUnderlyingError=0x7f8812a792a0 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
** EXPORT FAILED **```
jakub
2018-08-06 12:12
hmmmmm, eh?
jakub
2018-08-06 12:13
looks like a file that is in Jenkins home...
jacqueswww
2018-08-06 12:15
cool thanks :wink:
jakub
2018-08-06 12:15
after the dev meeting
jakub
2018-08-06 12:27
oh I see, this build used `macos2` host that Roman fixed
jakub
2018-08-06 12:27
so our build depends on some files in jenkins home dir
jakub
2018-08-06 12:27
that's awful
jakub
2018-08-06 12:27
we need to fix this
nastya
2018-08-06 12:28
:thinking_face:
jakub
2018-08-06 12:28
let me see what file this is
nastya
2018-08-06 12:28
seems nightly job lives its own life - always something new after weekends
jakub
2018-08-06 12:29
all software that moves fast does
nastya
2018-08-06 12:29
:sweat_smile:
jakub
2018-08-06 12:30
okay, this file seems to contain status Apple team ID
jakub
2018-08-06 12:30
I don't know if this is sensitive info, I assume it is
jakub
2018-08-06 12:31
I guess we could store it in Jenkins and then template the file in Jenkinsfile
volodymyr.kozieiev
2018-08-06 12:33
@maxr, @tatiana, sorry, didn't see this thread and already merged PR
jakub
2018-08-06 12:33
that's fine, I'll include it in my PR
jakub
2018-08-06 12:33
after dev meeting
jakub
2018-08-06 12:43
The Team ID doesn't seem like it's a secret
jakub
2018-08-06 12:43
it's already here:
jakub
2018-08-06 12:43
so I ca just put those files in the repo
nastya
2018-08-06 12:43
i have no idea what team id is
jakub
2018-08-06 12:43
it's an Apple internal thing used when uploading the app
jakub
2018-08-06 12:55
heres the fix
nastya
2018-08-06 12:56
:raised_hands::cookie::coffee:
jakub
2018-08-06 12:56
I really need some breakfast
nastya
2018-08-06 12:56
sure !
jakub
2018-08-06 12:56
waking up early is not my thing
nastya
2018-08-06 12:56
i am sorry (
jakub
2018-08-06 12:57
it's cool, the dev meeting was what got me out of bed :smile:
jakub
2018-08-06 13:50
okay
jakub
2018-08-06 13:50
which `node-07`?
jakub
2018-08-06 13:50
and on what timescale?
jakub
2018-08-06 13:51
7 days
jakub
2018-08-06 13:51
yeah, the `do-ams3` one seems a bit higher than others
jakub
2018-08-06 13:53
seems most cpu usage come from the whisper containers
jakub
2018-08-06 13:54
sure sure why though
jakub
2018-08-06 13:55
seems like theres a spike of peers for this host, from ~10 to ~20
jakub
2018-08-06 13:55
in the past ~2 days
jakub
2018-08-06 13:55
maybe that's it?
jakub
2018-08-06 13:55
and about geth inbout/outbound metrics
jakub
2018-08-06 13:56
I don't know, they are marked in legend as `[B/s]` so I would assume bytes
jakub
2018-08-06 13:57
i've updated the axes
jakub
2018-08-06 13:57
but I'm not 100% sure
jakub
2018-08-06 13:57
the name of the metric does not indicate unit
jakub
2018-08-06 13:57
`geth_discv5_inboundTraffic_overall`
jakub
2018-08-06 13:57
maybe someone from the team can tell us
divan
2018-08-06 14:02
@jacqueswww node07 was running other version of statusd
divan
2018-08-06 14:02
I used this server to test memory leak solution.
divan
2018-08-06 14:03
Memory leak seems to be solved, but I also noticed slightly higher CPU usage. Didn't check it few days. Is it bad now?
jakub
2018-08-06 14:03
Oh right, and it was running with ppref
jakub
2018-08-06 14:03
Maybe ppref is reaponsible
divan
2018-08-06 14:03
I think it has pprof disabled now, but I'm not sure.
divan
2018-08-06 14:04
Might be, yes
jakub
2018-08-06 14:04
I'm going to get breakfast now, I'll check in 30 min
jacqueswww
2018-08-06 14:45
ah ok :wink:
jacqueswww
2018-08-06 14:46
probably ppref then
jakub
2018-08-06 14:57
```admin@node-07.do-ams3.eth.beta:~ % d inspect statusd-whisper-1 | grep pprof
"-pprof",
"-pprofport=1337",
"-pprof",
"-pprofport=1337"```
jakub
2018-08-06 14:57
yup
jakub
2018-08-06 14:58
shall I turn it off?
adamb
2018-08-06 15:27
@divan ^^^
divan
2018-08-06 16:00
Not yet
divan
2018-08-06 16:00
I still see some memory increase, would love to debug a bit more
jakub
2018-08-06 16:01
cool
jakub
2018-08-06 16:01
just tell me when you're done
nastya
2018-08-07 07:42
we still dont have nightly build `[nightly] Running shell script
+ xcodebuild -exportArchive -exportPath status_appstore -archivePath status.xcarchive -exportOptionsPlist /Users/jenkins/archive-release.plist
error: Couldn't load -exportOptionsPlist: The file ?archive-release.plist? couldn?t be opened because there is no such file.
Error Domain=NSCocoaErrorDomain Code=260 "The file ?archive-release.plist? couldn?t be opened because there is no such file." UserInfo={NSFilePath=/Users/jenkins/archive-release.plist, NSUnderlyingError=0x7fb168306000 {Error Domain=NSPOSIXErrorDomain Code=2 "No such file or directory"}}
** EXPORT FAILED **` https://ci.status.im/job/status-react/job/nightly/543/
igor680
2018-08-07 08:20
we have a PR for that, but it wasn't merged
nastya
2018-08-07 08:20
is it possible to merge it?
igor680
2018-08-07 08:21
nope, merge conflicts
igor680
2018-08-07 08:21
I pinged Jacob
andrey
2018-08-07 08:54
@sergii ^
andreaf
2018-08-07 09:42
I can't log in to https://canary.status.im/ with github. do I need to be added?
andreaf
2018-08-07 09:45
/cc @pedro
jakub
2018-08-07 11:17
Sure, but I have to check if I have access
jakub
2018-08-07 11:29
oh yeah, I have access as `root`
jakub
2018-08-07 11:34
hmmm, question
jakub
2018-08-07 11:35
what domain should I use for this one?
jakub
2018-08-07 11:35
unless you want a different subdir
andrey
2018-08-07 11:43
@sergii?
sergii
2018-08-07 11:47
Hey, `faucet-rinkeby` as subdomain for Rinkeby sounds reasonable for me.
So https://faucet-rinkeby.status.im/.
jakub
2018-08-07 11:47
okay
jakub
2018-08-07 11:49
Oh wait, I checked wrong IP, I can't access `51.15.60.23`
jakub
2018-08-07 11:49
@adamb can you add my key?
jakub
2018-08-07 11:49
if you have access
pedro
2018-08-07 11:58
I'm seeing
```
403 Permission Denied
Invalid Account
```
cc @jakub
adamb
2018-08-07 12:25
I will be back in 10 mins
andrey
2018-08-07 14:19
@jakub @adamb any updates on this ? :slightly_smiling_face:
jakub
2018-08-07 14:19
I need access to that host =]
adamb
2018-08-07 14:19
oh sorry
adamb
2018-08-07 14:19
checking
adamb
2018-08-07 14:19
it's `51.15.60.23`??
adamb
2018-08-07 14:20
if yes, I do not have access :disappointed:
jakub
2018-08-07 14:20
hmmm, I guess I can set up the proxy on a different host
adamb
2018-08-07 14:21
@divan might be our last resort
divan
2018-08-07 14:55
No, I never touched faucet
divan
2018-08-07 14:55
Or do you say that I might have access
divan
2018-08-07 14:55
?
jakub
2018-08-07 14:55
you might have access?
divan
2018-08-07 14:55
I'm from phone now
divan
2018-08-07 14:55
Will be at laptop in 3 hours
jakub
2018-08-07 14:56
no problem
jakub
2018-08-07 14:56
i think I'll just setup a proxy on the previous host
jakub
2018-08-07 14:56
but after I'm done fixing nightlies
jakub
2018-08-07 16:19
okay, nightly works(at least on macos1)
jakub
2018-08-07 16:19
I'll test `macos2` now
jakub
2018-08-09 03:35
I used the `51.15.45.169` host to setup the proxy since I can't access `51.15.60.23`
jakub
2018-08-09 03:35
It's not a permanent solution but it's enough for nomw
nastya
2018-08-09 06:58
anything new regarding nightly? we still dont have it
andrey
2018-08-09 07:12
thanks Jacob! @sergii could you make changes in your PR ? to use `https://faucet-rinkeby.status.im/faucet-info` ?
sergii
2018-08-09 07:12
yes! Doing now.
anna
2018-08-09 07:44
why on https://status-im.github.io/nightly/ date is ?Updated: 2018/08/08 18:45:23 UTC? while last successful build on https://jenkins.status.im/job/status-react/job/nightly/ was on August 7?
anna
2018-08-09 08:28
@jakub @igor680 build https://ci.status.im/job/status-react/job/nightly/563/console with merged https://github.com/status-im/status-react/pull/5427 failed ```** ARCHIVE FAILED **
The following build commands failed:
CodeSign /Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-ejyivsstnbyceeegdbpzgbjrzddm/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/RSKImageCropper.framework
CodeSign /Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-ejyivsstnbyceeegdbpzgbjrzddm/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/InstallationBuildProductsLocation/Applications/StatusIm.app/Frameworks/QBImagePicker.framework
(2 failures)``` so no nightly build since Monday that is a bit too much
igor680
2018-08-09 08:29
Something is wrong with the certs again :(
chad
2018-08-09 08:31
https://docs.fastlane.tools/actions/cert/
sweet way to manage :slightly_smiling_face:
igor680
2018-08-09 08:32
yeah, I think the time has come, it's total mess now
chad
2018-08-09 08:34
you won't regret it :heart:
jakub
2018-08-09 11:49
https://ci.status.im/job/status-react/job/nightly/563/consoleFull fails because it's on macos2
jakub
2018-08-09 11:49
I'll turn it off for now
jakub
2018-08-09 11:50
https://ci.status.im/job/status-react/job/nightly/564/consoleFull succeeded but failed because this: https://github.com/status-im/status-react/pull/5430 - was merged only an hour ago
jeluard
2018-08-09 11:51
Are PRs builds broken currently?
If so can this explain https://ci.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-5388/10/console ?
jakub
2018-08-09 11:51
hm
jakub
2018-08-09 11:51
let me see
jakub
2018-08-09 11:52
```+ lein test-cljs
Updating hook: pre-commit
Could not find artifact pluto:pluto:jar:iteration-2-SNAPSHOT in clojars (https://repo.clojars.org/)
This could be due to a typo in :dependencies, file system permissions, or network issues.
If you are behind a proxy, try setting the 'http_proxy' environment variable.
Error encountered performing task 'doo' with profile(s): 'test'
Could not resolve dependencies```
jakub
2018-08-09 11:52
eeee
jakub
2018-08-09 11:52
what?
jeluard
2018-08-09 11:53
Hmm ok my bad I was reading the XCode errors
jakub
2018-08-09 12:07
so I guess we'd want to codify the "Build (iOS)" step using fastlane? I can try but I have no idea how long it will take
chad
2018-08-09 12:10
I think it is pretty quick, providing you can get your head around the codesigning options but there are some great docs eg https://codesigning.guide
jakub
2018-08-09 12:11
well, i don't know i I can do just signing
jakub
2018-08-09 12:12
I've never worked with anything iOS related like that
jakub
2018-08-09 12:12
do I have to do the whole build in fastlane? can I just sign and unsigned app?
jakub
2018-08-09 12:12
how do I create andunsigned app?
jakub
2018-08-09 12:12
there's a lot of questions
jakub
2018-08-09 12:13
also, I just found something interesting in our jenkinsfile
jakub
2018-08-09 12:13
``` sh "security unlock-keychain -p ${password} login.keychain"```
jakub
2018-08-09 12:13
which uses ```
withCredentials([
string(credentialsId: 'jenkins_pass', variable: 'password')
]) {
```
jakub
2018-08-09 12:13
WHICH MEANS the password would be correct just for one host
jakub
2018-08-09 12:15
but no, this fails before that step
jakub
2018-08-09 12:15
on ``` sh 'xcodebuild -workspace ios/StatusIm.xcworkspace -scheme StatusIm -configuration release -archivePath status clean archive'
```
igor680
2018-08-09 12:23
but I guess code signing is broken on non macos1
jakub
2018-08-09 12:24
yeah
jakub
2018-08-09 12:24
worked fine here on `macos1`
jakub
2018-08-09 12:24
I'll do some tests to see if I can fix it the way Roman did before, but I'll start working on using `fastlane`
jakub
2018-08-09 12:24
because dealing with xcode is some kind of nightmare
igor680
2018-08-09 12:25
yeah, xcode, signing and CI is a bomb
chad
2018-08-09 12:25
For shits and giggles I?m playing with Fastlane :slightly_smiling_face:
jakub
2018-08-09 12:26
yup, succeeds fine the `Build (iOS)` stage on `macos1`:
https://ci.status.im/job/status-react/job/nightly/566/
I've disabled `macos2` for now
chad
2018-08-09 12:27
This seems to kick it off quite nicely
``` lane :testbuild do
build_app(
scheme: "StatusIm",
workspace: "ios/StatusIm.xcworkspace")
end```
jakub
2018-08-09 12:28
nice
jakub
2018-08-09 12:28
can you also add `-configuration release`?
chad
2018-08-09 12:29
all params
jakub
2018-08-09 12:31
sweet
jakub
2018-08-09 12:38
I think I have a fix for current setup
jakub
2018-08-09 12:39
we were using a password for `macos1` to unlock `macos2`
jakub
2018-08-09 12:39
I just tested this by hand on `macos2`
jakub
2018-08-09 12:39
https://github.com/status-im/status-react/pull/5442 - possible fix for broken code signing on `macos2`
igor680
2018-08-09 12:40
ohhhh
jakub
2018-08-09 12:40
also, the `security` command was too late
jakub
2018-08-09 12:40
after first `xcodebuild`
jakub
2018-08-09 12:41
that doens't mean we shouldn't work on using `fastlane`
jakub
2018-08-09 12:41
since xcode is all manual point-n-click setup
jakub
2018-08-09 12:43
ok, it's pushed
jakub
2018-08-09 12:43
and now I'm going to get breakfast
jakub
2018-08-09 12:43
hopefully by the time I'm done this will succeed
chad
2018-08-09 12:46
please get a bacon, egg and cheese on a toasted poppy bagel for me :slightly_smiling_face:
jakub
2018-08-09 12:46
will do
igor680
2018-08-09 12:47
but no onions!
jakub
2018-08-09 12:47
what the hell? onions are amazing
jakub
2018-08-09 12:47
what is wrong with you Igor?
chad
2018-08-09 12:48
bacon egg and cheese is new york rite of passage and best hangover cure too
jakub
2018-08-09 12:48
I found this great place
jakub
2018-08-09 12:48
run by a lovely couple from hong kong
jakub
2018-08-09 12:49
they make great sammiches
jakub
2018-08-09 12:49
like the one you mentioned
chad
2018-08-09 12:50
good reading for new yorkers
chad
2018-08-09 12:50
```The great virtue of the bacon, egg and cheese on a roll, or its variations, is in what it doesn't do. It doesn't divide New Yorkers by class, income or neighborhood. It doesn't seek publicity. It doesn't convey status or bragging rights. It just conveys nutrition and, if you need it, settles your nerves. It is a secret handshake that New Yorkers exchange, not with one another, but with the city.```
chad
2018-08-09 12:50
`a secret handshake that new yorkers exhange` :heart:
jakub
2018-08-09 12:58
>It doesn't seek publicity. It doesn't convey status or bragging rights.
No, that's done by the mass legacy media, politicians, and race husslers
jakub
2018-08-09 15:52
XCode signing settings on macos2... it's been klike that for 10 min.
igor680
2018-08-09 15:55
@jakub I want to fix that next week, to move everything to fastlane. Since I did setup this stuff before, it might be easier that way.
jakub
2018-08-09 15:56
yeah, I'd be happy to help in any way with the fastlane setup
jakub
2018-08-09 15:56
I think that Satan is testing me through this xcode signing stuff
jakub
2018-08-09 15:58
do you have any pointers on how to start with this?
jakub
2018-08-09 15:58
do we want to replace all the 3 `xcodebuild` calls in "Build (iOS)" stage with fastlane steps?
jakub
2018-08-09 15:59
would that be a single `fastlane` run or multiple?
jakub
2018-08-09 15:59
from what I understand this cleans the workspace:
``` sh 'xcodebuild -workspace ios/StatusIm.xcworkspace -scheme StatusIm -configuration release -archivePath status clean archive'
```
jakub
2018-08-09 15:59
then these two build two versions of the app:
```
sh 'xcodebuild -exportArchive -exportPath status_appstore -archivePath status.xcarchive -exportOptionsPlist ios/archive-release.plist'
sh 'xcodebuild -exportArchive -exportPath status -archivePath status.xcarchive -exportOptionsPlist ios/archive-develop.plist'
```
jakub
2018-08-09 15:59
one for release and one for development
jakub
2018-08-09 15:59
not sure what's the difference
jakub
2018-08-09 16:00
I guess the release setup adds these settings:
```
<key>uploadBitcode</key>
<false/>
<key>compileBitcode</key>
<true/>
<key>ITSAppUsesNonExemptEncryption</key>
<false/>
```
jakub
2018-08-09 16:02
although both otuput to `status.xcarchive`... so why do we run both?
igor680
2018-08-09 16:02
Don?t bother now, there are multiple fastlane utils needs to be used
jakub
2018-08-09 16:02
okay, though I'd still like to know why we run two builds
jakub
2018-08-09 16:03
one for `ios/archive-develop.plist` and one for `ios/archive-release.plist`
jakub
2018-08-09 16:03
(interestingly enough the one for `ios/archive-develop.plist` succeeds on `macos2`)
igor680
2018-08-09 16:03
what job is that?
jakub
2018-08-09 16:03
`Jenkinsfile.nightly_fastlane`
jakub
2018-08-09 16:03
the one used in the `nightly`
igor680
2018-08-09 16:04
No idea, It looks as weird to me too
jakub
2018-08-09 16:04
yeah
jakub
2018-08-09 16:05
interestingly enough `Jenkinsfile.upload_release_ios` uses just `ios/archive-release.plist` while `Jenkinsfile.release` uses `ios/archive-develop.plist`
igor680
2018-08-09 16:08
okay, but that is a local file to a node, right?
jakub
2018-08-09 16:08
it's in the repo
jakub
2018-08-09 16:08
jakub
2018-08-09 16:08
igor680
2018-08-09 16:10
ah, okay
the `app-store` method creates builds that only can be used for test flight or app store submissions
jakub
2018-08-09 16:10
oh
igor680
2018-08-09 16:10
so `develop` you need if you need to distribute it differently
jakub
2018-08-09 16:11
I see
igor680
2018-08-09 16:11
but I have no clue why we might need both
igor680
2018-08-09 16:11
do we upload nightly both to diawi and to TF?
jakub
2018-08-09 16:11
heh, that's fine
jakub
2018-08-09 16:12
as I said, I'm all for setting up fastlane for this, I just know very little about this ios build process
jakub
2018-08-09 16:12
I'll help however I cna
igor680
2018-08-09 16:12
so, I just need VNC access to these machines
igor680
2018-08-09 16:12
I can fix that and then document what I did
jakub
2018-08-09 16:12
sure
jakub
2018-08-09 16:13
I can PM you the password
jakub
2018-08-09 16:13
or I can give you access to the `infra-pass` repo
jakub
2018-08-09 16:13
it has all the cool stuff you might need
igor680
2018-08-09 16:14
nah, the less I know the better
jakub
2018-08-09 16:14
fine
lukasz949
2018-08-13 09:47
Hi! I see `no space left on device, write` in a jenkins build on ?iOS build? stage: https://ci.status.im/job/status-react/job/pull%20requests/job/PR-5319/18/
Should I restart it?
igor680
2018-08-13 10:13
you can try
lukasz949
2018-08-13 11:36
I tried but there was another error about fetching the repo :open_mouth: Trying again
tatiana
2018-08-15 13:14
@jakub or @igor680 PR build for IOS failed again, is it infrastracture issue?
https://ci.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-5377/8/console
```
The following build commands failed:
PhaseScriptExecution Download\ Core /Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-aqauveyoszyqdsgpsfimhwdgknlt/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/IntermediateBuildFilesPath/RealmJS.build/Release-iphoneos/RealmJS.build/Script-F63FF2C51C12462600B3B8E0.sh
(1 failure)
[13:38:21]: Exit status: 65
```
Appreciate your help!
igor680
2018-08-15 13:16
That is a realm issue, it happens sometime, but I didn't manage to reproduce it yet. Rebasing PR to the latest develop and restarting might help, because it isn't persistent.
igor680
2018-08-15 13:30
reproduced it locally!
it happens when 2 ios builds are building on the same runner
igor680
2018-08-15 13:30
at the same time
tatiana
2018-08-15 13:51
Ok, so I should just restart it?
igor680
2018-08-15 13:52
yeah
adamb
2018-08-15 23:00
Should we cancel previous PR builds if a new commit is pushed? Sometimes I can see like 3-4 builds related to a single PR in the Jenkins queue which seems like unnecessary waisting of resources?
igor680
2018-08-16 06:16
Yeah, we should. But last time I looked into this, it wasn't trivial :(
adamb
2018-08-16 06:20
This plugin that we use does not seem really configurable :(
igor680
2018-08-16 12:58
@jakub jfyi I put "use this node as much as possible" to all our mac nodes to allow desktop builds to use them
jakub
2018-08-16 12:58
sweet
roman
2018-08-17 06:15
igor680
2018-08-17 07:20
yep, one more typo in Jenkinsfiles!
fix https://github.com/status-im/status-react/pull/5538
igor680
2018-08-17 07:20
thanks :slightly_smiling_face:
igor680
2018-08-17 08:28
next version of realm shouldn't have this problem with parallel builds: https://github.com/realm/realm-js/pull/1968
andrey
2018-08-17 08:58
`ENOSPC: no space left on device, write` on Android build
andrey
2018-08-17 09:00
@jakub?
andrey
2018-08-17 09:41
@igor680?
igor680
2018-08-17 09:41
let me see
igor680
2018-08-17 09:48
cleaning up started
igor680
2018-08-20 08:43
@jakub sorry for bad news early in the week... we have 2 problems with our builds now
igor680
2018-08-20 08:43
one:
```
Running on macos1 in /Users/jenkins/workspace/react_pull_requests_PR-5534-EWBJEIAQBA72R5G3MARVUM3BWIQOXHLFK34CZBTCZ6FPQRMXOUGQ
<...>
* What went wrong:
A problem occurred configuring project ':app'.
> Failed to install the following SDK components:
[Android SDK Platform 27, Android SDK Build-Tools 26.0.2]
The SDK directory (/usr/local/share/android-sdk) is not writeable,
please update the directory permissions.
```
igor680
2018-08-20 08:44
two:
we show only desktop build results on github PRs
andrey
2018-08-20 09:48
@adamb in nightly
andrey
2018-08-20 09:49
on Android only
andrey
2018-08-20 09:51
@sergii @nastya was android tested in PR?
sergii
2018-08-20 09:52
Sure it was.
andrey
2018-08-20 09:53
kk thanks, just to make sure
andrey
2018-08-20 09:56
@adamb @igor680 node 4afd9e6c
andrey
2018-08-20 09:57
looks like it's a right commit
adamb
2018-08-20 10:06
`v0.11.0` is exactly that commit
andrey
2018-08-20 10:07
@adamb any thoughts why this error on Android may be?
adamb
2018-08-20 10:08
which error?
adamb
2018-08-20 10:10
so on iOS it works but not on android?
nastya
2018-08-20 10:11
android 8 i guess
adamb
2018-08-20 10:13
and other Androids work?
nastya
2018-08-20 10:14
8 and 9 does not - 7 works
adamb
2018-08-20 10:14
interesting
adamb
2018-08-20 10:14
but other bindings do work on 8 and 9?
andrey
2018-08-20 10:17
@sergii what Android did you test on?
sergii
2018-08-20 10:24
Hey, so this issue is on Android 7 too (on latest nightly)
Wallet refactoring tested on Android 7
andrey
2018-08-20 10:26
so nightly build is somehow different from PR builds?
igor680
2018-08-20 10:43
we can re-build the PR and see if that issue is there
sergii
2018-08-20 10:45
I've checked nightly before wallet refactor is merged and it's OK there.
Also I checked some very latest PR build with node 4afd9e6c and it's all fine there too (can sign message, can sign transaction)
igor680
2018-08-20 10:47
yeah, but the point is re-building that
anna
2018-08-20 11:11
@igor680 what is strange is that on ?staging fleet nightly build? https://status-im.slack.com/archives/C88MZP11P/p1534762230000100 there is no crash on Android if send ETH (all is fine there)
igor680
2018-08-20 11:19
can we rebuild a nightly?
igor680
2018-08-20 11:19
I know @jakub was doing a lot there, so maybe it was just a wrong build?
jakub
2018-08-20 11:29
i know, im working on it
jakub
2018-08-20 11:30
yeah, just run chmod on the dir
jakub
2018-08-20 11:31
done
anna
2018-08-20 12:34
looks good on new nightly build - can send ETH from wallet fine on Android :shrug:
nastya
2018-08-20 12:34
�\_(?)_/�
adamb
2018-08-20 12:36
@jakub I cannot Sign in on https://prometheus.status.im/
```
403 Permission Denied
redirect_uri_mismatch
```
jakub
2018-08-20 12:36
checking
nastya
2018-08-20 12:37
@anna can you sign a message to cryptokitties on android 9 for me please?
jakub
2018-08-20 12:37
please check now
jakub
2018-08-20 12:37
github was using `http` instead of `https` in one of the urls, no idea why
adamb
2018-08-20 12:38
works, thanks!
adamb
2018-08-20 12:38
so we now use `http`?
jakub
2018-08-20 12:38
no, https
adamb
2018-08-20 12:38
ok
jakub
2018-08-20 12:38
http i how we started off in the beginning
adamb
2018-08-20 12:38
ah ok
anna
2018-08-20 12:40
can sign (didn't check sending tx)
anna
2018-08-20 12:40
so sign - ok, purchase of new kitty - didn't try
nastya
2018-08-20 12:40
great thats enough thank you :heart:
igor680
2018-08-20 12:45
Yeah, so it was some broken build from Jenkinsfiles refactoring
nastya
2018-08-20 12:49
@jakub @igor680 jenkins queue is insane again because of desktop
jakub
2018-08-20 12:53
i know
jakub
2018-08-20 12:53
theres something about desktop pr builds which is wrong
nastya
2018-08-20 12:57
:okay:
jakub
2018-08-20 12:57
they scan git repos every 24 hours
jakub
2018-08-20 12:57
and for some reason start builds for PRs that havent changed
jakub
2018-08-20 12:57
no idea why
andrey
2018-08-20 16:02
hey @jakub any ideas why this commit reverted my changes https://github.com/status-im/status-im.github.io/commit/0055726f1704439bfa38d8e0eb0cb4d0849b374c ?
sergii
2018-08-21 12:17
hey @jakub the recent nightly job fails with `Google Api Error: apkUpgradeVersionConflict: APK specifies a version code that has already been used.` (https://ci.status.im/job/status-react/job/nightly/593/). Could you help to understand how can we fix it, please?
jakub
2018-08-21 12:18
yeah, I guess some race condition with one of my combined builds
jakub
2018-08-21 12:18
it should work if you re-run it
jakub
2018-08-21 12:18
as long as it fetches new tags, and `build_no.sh` does, then it should be fine
sergii
2018-08-21 12:19
good, let me push once again
nastya
2018-08-22 11:01
android build fails
nastya
2018-08-22 11:01
```FAILURE: Build failed with an exception.
* Where:
Build file '/Users/jenkins/workspace/react_pull_requests_PR-5494-TZ2PKRN3OWAFFA6LW7NRPTQQYLZODA4EK4GAOPBTGV64UXAVS2GA/android/app/build.gradle' line: 105
* What went wrong:
A problem occurred evaluating project ':app'.
> For input string: ""
* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output. Run with --scan to get full insights.
* Get more help at https://help.gradle.org
BUILD FAILED in 5s```
nastya
2018-08-22 11:02
is it something you can assist with @igor680?
igor680
2018-08-22 11:06
It is super weird, I've seen this issue but it went away as soon as I rebuilt. Can you send a link to the failed job there?
igor680
2018-08-22 11:07
Okay, sure. I will take a look at that in an hour or so.
igor680
2018-08-22 12:30
@nastya okay, builds should feel better now
nastya
2018-08-22 12:30
:+1:
roman
2018-08-22 16:40
> No space left on device
https://ci.status.im/job/status-react/job/desktop/job/pull-requests/job/PR-5606/2/console
nastya
2018-08-23 13:26
whats wrong with diawi ? builds download is so slow
nastya
2018-08-23 13:28
@jakub?
jakub
2018-08-23 13:28
is it? well, Diawi is an external service, I don't now whats wrong with them
dmitry.shulyak
2018-08-23 13:30
btw my speed for downloading from diawi is fine
nastya
2018-08-23 13:33
arent u managing everythin? :sweat_smile:
jakub
2018-08-23 13:34
I wish, you're thinking of Bogdanoff Twins
nastya
2018-08-23 13:34
yes sir
tatiana
2018-08-28 11:50
@jakub
```
stdout:
stderr: error: could not lock config file .git/config: No space left on device ```
- at attempt to build
https://ci.status.im/job/status-react/job/desktop/job/pull-requests/job/PR-5663/.
Can u help please?
jakub
2018-08-28 11:52
yeah
jakub
2018-08-28 11:52
should be fine now
jakub
2018-08-28 12:02
today I'd like to start using the new combined parallel build for PRs, of course as I do the might be some issues. Please report to me any breakage in Jenkins PR builds so I can patch it
yenda
2018-08-28 12:54
I noticed last night that jenkins doesn't report mobile builds anymore, at first I was confused because it reports desktop and I didn't understand why the build was finished without building mobile, now that mobile builds are separated
jakub
2018-08-28 12:55
yeah, it shows the desktop ones, not sure how it decides which, but hopefully today is the day
jakub
2018-08-28 12:55
that we get rid of the split builds
dmitryn
2018-08-28 13:03
tatiana
2018-08-28 13:42
tatiana
2018-08-28 14:17
@jakub can you resolve it?
jakub
2018-08-28 14:18
that's 2 hours old
jakub
2018-08-28 14:18
it's been reoslved
dmitryn
2018-08-28 14:22
This one seems not resolved https://ci.status.im/blue/organizations/jenkins/status-react%2Fdesktop%2Fpull-requests/detail/PR-5665/2/pipeline
jakub
2018-08-28 14:22
let me see
jakub
2018-08-28 14:23
what's wrong with it?
dmitryn
2018-08-28 14:24
`No space left on device`
jakub
2018-08-28 14:25
ah, macos1 this time
jakub
2018-08-28 14:25
why do they grow so fast, it's weird
jakub
2018-08-28 14:25
check now
dmitryn
2018-08-28 17:05
We have issues building https://ci.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-5610/19/console
Checking console output, not clear what went wrong
roman
2018-08-28 19:02
https://github.com/status-im/status-react/pull/5606 have problems with build. It hasn?t been started
dmitryn
2018-08-29 10:44
https://github.com/status-im/status-react/pull/5610 cannot build
https://ci.status.im/blue/organizations/jenkins/status-react%2Fpull%20requests/detail/PR-5610/16/pipeline
```
java.lang.IllegalArgumentException: Null value not allowed as an environment variable: BUILD_TYPE
```
jeluard
2018-08-29 11:41
Are Jenkins build supposed to take 2h30? :sweat_smile:
https://ci.status.im/job/status-react/job/pull%20requests/job/PR-5686/
igor680
2018-08-29 12:15
Nope, fixing them now
george952
2018-08-29 14:29
Alright, so we have a certificate, a private key, a way to unlock the keychain, import everything and codesign the macOS application bundle from the CLI. What goes where / what next? @jakub @igor680
jeluard
2018-08-29 14:45
We have a build queue of 54 builds on Jenkins..
jeluard
2018-08-29 14:46
Can we clean this up? We need a build pretty quick to test a PR to be included in next release
jeluard
2018-08-29 14:47
If no one yell I will just kill builds randomly
igor680
2018-08-29 14:49
Yelling
igor680
2018-08-29 14:49
Bad luck but we need this queue to get through
jeluard
2018-08-29 14:50
Why?
igor680
2018-08-29 14:51
because we need the old unrebased prs to finally pass
igor680
2018-08-29 14:51
Otherwise they would screw up Jenkins config again
igor680
2018-08-29 14:52
and you will have random failures in normal prs
jeluard
2018-08-29 14:53
Even if those old PRs don't run at all?
igor680
2018-08-29 14:53
They will run eventually
igor680
2018-08-29 14:53
because of Jenkins scanning the branches
jeluard
2018-08-29 14:54
As long as my PR is before I'm fine then! And I can kill fast
igor680
2018-08-29 14:55
and yeah, `.combined<blah>` might be parts of your PR so be careful
jeluard
2018-08-29 14:56
This jenkins thing is becoming seriously complex
igor680
2018-08-29 14:59
it is
adan
2018-08-29 15:01
then I am going to disable https://ci.status.im/job/end-to-end-tests/job/status-app-nightly/ since each PR is triggering the build
igor680
2018-08-29 15:02
it should only happen for nightlies, right?
adan
2018-08-29 15:02
yep
adan
2018-08-29 15:03
I believe that this is happening due to `missing rebase for those PRs`
adan
2018-08-29 15:03
as I wrote in the thread above, but I am might be wrong
igor680
2018-08-29 15:04
ah, ok
george952
2018-08-29 16:30
This is what the scripts (initial key deployment and signing) will look like:
https://gist.github.com/gnl/755f6d76716da8f52dc638d9b96451be
george952
2018-08-29 19:45
What do we do for CI/deployment dependencies from NPM, where do we put them? In the project dependencies doesn't seem right.
EDIT: For the record/documentation, I installed appdmg as an optional, OS-specific dependency:
https://github.com/status-im/status-react/pull/5702
maxr
2018-08-29 20:34
could you please provide example of some dependency which we have and where it's placed currently and what should work better? Thanks
george952
2018-08-29 21:52
Not really, I'm sort of looking for examples myself. I need a new deployment dependency ? https://github.com/LinusU/node-appdmg ? in order to create a nice and easy to use .dmg macOS bundle and I'd rather not add it to the project dependencies and have it end up in the runtime JS bundle as it's not needed there.
maxr
2018-08-29 23:47
Should it be just part of some build-deployment script (currently we have Jenkinsfile which takes this kind of job, but build-deployment code should migrate to some kind of Makefile, which will be triggered by Jenkins or by developer locally, finally) ?
tatiana
2018-08-30 07:16
tatiana
2018-08-30 07:29
@jakub @igor680 to let you know - PR builds still don't work for me:
1) https://ci.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-5318/4/ - on Android, IOS, Linux build with error:
```
java.lang.IllegalArgumentException: Null value not allowed as an environment variable: BUILD_TYPE
```
2) https://ci.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-5664/8/ on Mac OS, Linux builds with error:
```
[ 96%] Building CXX object bin/CMakeFiles/StatusIm.dir/__/main.cpp.o
/Users/jenkins/workspace/status-react/combined/desktop-macos/desktop/main.cpp:29:10: fatal error: 'exceptionglobalhandler.h' file not found
#include "exceptionglobalhandler.h"
^~~~~~~~~~~~~~~~~~~~~~~~~~
1 error generated.
make[2]: *** [bin/CMakeFiles/StatusIm.dir/__/main.cpp.o] Error 1
make[1]: *** [bin/CMakeFiles/StatusIm.dir/all] Error 2
```
Appreciate any help, thanks.
igor680
2018-08-30 07:44
First looks like it isn't rebased
The second ? does it build locally?
tatiana
2018-08-30 07:59
First looks like it isn't rebased - how it can be rebased? Should I ask dev manually rebase it?
Second - I just trigger build manually in Jenkins
maxr
2018-08-30 08:02
Second error is related to my last merged commit tonight. But it doesn't appear on automatic PR builds (I've resolved it there)
tatiana
2018-08-30 08:04
Thanks, Max.
OK, what should I do to get 4 working builds for all platforms in 1 and 2 case?
maxr
2018-08-30 08:07
I will try to rebase https://github.com/status-im/status-react/pull/5664 and force push into PR to see if it will still fail or not.
Regarding changes in https://github.com/status-im/status-react/pull/5664 itself, it seems Desktop should be updated as well for newer status-go version
tatiana
2018-08-30 08:10
yes, thats why I want to check it manually.
maxr
2018-08-30 08:15
Have rebased both PRs. Please let me know if any of them still fails, thanks
adan
2018-08-30 09:30
@jakub, @igor680 Morning, do we have `Upload apk for e2e tests` (e.g. https://github.com/status-im/status-react/blob/develop/ci/Jenkinsfile#L102) step in the combined job?
adan
2018-08-30 09:40
for exmaple in https://ci.status.im/blue/organizations/jenkins/status-react%2Fpull%20requests/detail/PR-5668/7/pipeline/116/ I can see that e2e apk has been uploaded only to digital ocean
adan
2018-08-30 09:41
however I might be looking into the wrong place
adan
2018-08-30 10:08
ignore the question, I've found that under Android stage
igor680
2018-08-30 10:16
So, you you need the link to sauce labs instead published?
adan
2018-08-30 10:19
nope, there is no link to Sauce
adan
2018-08-30 10:19
igor680
2018-08-30 10:19
Yeah, I know, I?m asking if you need that link instead
adan
2018-08-30 10:20
it is uploading PR jobs like im.status.ethereum-e2e-${GIT_COMMIT.take(6)}.apk
adan
2018-08-30 10:20
but we expect pr_number.apk
adan
2018-08-30 10:20
I will rework this by myself in e2e tests and probot
igor680
2018-08-30 10:20
Sure, ping me when the PR is ready
adan
2018-08-30 10:21
sure
adan
2018-08-30 10:21
or it's okay to use env.CHANGE_ID in https://github.com/status-im/status-react/blob/76e9501541197de74c4650889adaf8bc8f1127b5/ci/android.groovy#L40 ?
adan
2018-08-30 10:21
if there is env.CHANGE_ID
igor680
2018-08-30 10:26
Yeah, is it's only for e2e
jakub
2018-08-30 10:43
Yup, gotta rabase from develop
adan
2018-08-30 10:50
please do not move/merge the PR, I want to test it through all flow, including triggering tests by probot when the PR is in `TO TEST`
goranjovic
2018-08-30 11:43
@jakub @igor680 @anna Do we still have this issue and do we have any clue what was wrong? https://github.com/status-im/status-react/issues/5563
adan
2018-08-30 11:44
btw, we still suffering from some PRs which trigger e2e tests for nightly job, Is this is due to missing rebase for those PRs?
jakub
2018-08-30 11:44
I guess my question would be when did it start?
jakub
2018-08-30 11:44
which is the last working nightly?
jakub
2018-08-30 11:44
I'd assume so
adan
2018-08-30 11:45
Is there a way to rebase them automatically?
jakub
2018-08-30 11:45
jakub
2018-08-30 11:45
no
jakub
2018-08-30 11:45
can you show me an example of an e2e test triggered by a pr?
jakub
2018-08-30 11:45
if you know that you know the pr, if you know the pr you know the pr branch, if you know the pr branch you can pull it, rebase it, and push it
adan
2018-08-30 11:46
and the next one
adan
2018-08-30 11:47
I will disable the project for now
jakub
2018-08-30 11:47
i cant rebase that one
jakub
2018-08-30 11:47
`jinhojang93:patch-1`
jakub
2018-08-30 11:47
it's a user branch
jakub
2018-08-30 11:47
they have to do it
jakub
2018-08-30 11:47
I'll post in that PR
jakub
2018-08-30 11:48
this one i can fix
adan
2018-08-30 11:49
no worries, no need to rebase them manually
adan
2018-08-30 11:49
it's okay for me to run the nightly job once per day for several days until those PRs are gone
adan
2018-08-30 11:50
such approach takes much less time then searching and rebasing
adan
2018-08-30 11:50
I just wanted to be sure that my assumption is right
george952
2018-08-30 11:55
@maxr Yeah, I put this into Jenkinsfile and desktop.groovy for now, but I need to know where to put the executable that I'm calling:
https://github.com/status-im/status-react/blob/62f51eb7af9f0f3b22593ba79e93febb7c038bfa/ci/desktop.groovy#L187
It's on NPM and come to think about it, it's probably best to put it in the desktop's package.json devDependencies section and since it's not required by the project it won't have any effect on the final JS bundle.
@jakub - can we get rid of the Jenkinsfiles already in favour of the new config to avoid duplication/confusion? Right now I'm still copy-pasting between the two, cause I'm not entirely sure which one is used where.
jakub
2018-08-30 11:56
yeah, I can do that, though it's not entirely ready
george952
2018-08-30 11:56
No pressure, whenever it won't break anything.
jakub
2018-08-30 12:01
I'm sure some people will not like this
adan
2018-08-30 12:02
@jakub env.CHANGE_ID is null for a PR https://github.com/status-im/status-react/pull/5707 according to https://ci.status.im/job/status-react/job/combined/job/mobile-android/760/console (search `android saucelabs`)
it was working before in the old setup and right now it is null, maybe due to combined job, is there a way to get a PR number somehow?
jakub
2018-08-30 12:02
don't know
jakub
2018-08-30 12:02
will have to research
jakub
2018-08-30 12:04
i'm busy right now, please research yourself
jakub
2018-08-30 12:04
i'm sure there's some groovy script that can make it happen
adan
2018-08-30 12:09
@jakub and @igor680 please, add me to review next time if you are making significant changes which can impact e2e tests, so in future we can avoid such issues
lukasz949
2018-08-30 12:37
Can we do something about:
```
Still waiting to schedule task
Waiting for next available executor on 'macos'
```
I really see it quite often. Maybe auto scale infra when there is demand? :money_with_wings:
andrey
2018-08-30 13:12
build is failed, how i can find a reason?
dmitryn
2018-08-30 13:51
For example, here is failure https://ci.status.im/job/status-react/job/combined/job/mobile-ios/390/console
andrey
2018-08-30 13:52
ok thanks
andrey
2018-08-30 13:52
found the reason
```
Compiling ["index.desktop.js"] from ["components/src" "react-native/src/cljsjs" "react-native/src/desktop" "src" "env/prod"]...
[31mWARNING[0m: Extending an existing JavaScript type - use a different symbol name instead of js/String e.g string at line 246 target/desktop-prod/multihash/core.cljc
Error encountered performing task 'cljsbuild' with profile(s): 'prod'
```
andrey
2018-08-30 13:53
i added new library in this PR
jeluard
2018-08-31 07:45
@george952 Make sure you are public there: https://github.com/orgs/status-im/people?page=2
jeluard
2018-08-31 07:47
See previous link, you are private. This is your issue
george952
2018-08-31 07:48
Oh.
george952
2018-08-31 07:48
It's still a bit early for me. :smile: Thanks. But this must be a new thing, cause I successfully merged before.
george952
2018-08-31 07:50
Worked like a charm, thanks @jeluard
jeluard
2018-08-31 07:51
:+1::skin-tone-2:
nastya
2018-08-31 08:11
anyone can take a look at nightly job? It is failing to build https://ci.status.im/job/status-react/job/nightly/ with `Error signal (self time 5ms)`
george952
2018-08-31 08:29
@jakub @igor680
I want to bump the codesigning issue, cause the bundle compatibility PR has been merged:
https://github.com/status-im/status-react/pull/5668
And now we just need to get the script above into CI to improve the installation/upgrade experience on the desktop a lot. And to look better out there what with being a security-oriented shop shipping unsigned builds.
This'll likely require some back and forth so it'd be great if we could set aside some time for this whenever you guys are available. Thanks!
igor680
2018-08-31 08:33
Let?s do it next week?
george952
2018-08-31 08:56
@igor680 No problem, let's.
nastya
2018-08-31 09:03
anyone can explain ```This XML file does not appear to have any style information associated with it. The document tree is shown below.
<Error>
<Code>NoSuchKey</Code>
<BucketName>status-im</BucketName>
<RequestId>tx000000000000013f99d67-005b890434-a28f91-ams3a</RequestId>
<HostId>a28f91-ams3a-ams3</HostId>
</Error>``` ?
nastya
2018-08-31 09:05
I can't download any build from PR links , for ex https://github.com/status-im/status-react/pull/5712. It was alive yesterday
igor680
2018-08-31 09:48
Probably something is wrong on Digital Ocean side
pedro
2018-08-31 13:14
you need to replace `.` with `-` in the file name
nastya
2018-08-31 13:14
:+1:
pedro
2018-08-31 13:14
they were renamed yesterday
tatiana
2018-08-31 17:25
@jakub @jeluard new failure of builds due to https://github.com/status-im/status-react/pull/5388 (as @vitaliy said)
All mobile builds are failed (https://ci.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-5721/6/)
Error:
```
--- a/src/status_im/ui/screens/extensions/events.cljs
+++ b/src/status_im/ui/screens/extensions/events.cljs
[036m@@ -1,7 +1,7 @@[0m
(ns status-im.ui.screens.extensions.events
(:require [re-frame.core :as re-frame]
[031m- [pluto.registry :as registry] [0m
[031m- [status-im.utils.handlers :as handlers] [0m
[032m+ [pluto.registry :as registry][0m
[032m+ [status-im.utils.handlers :as handlers][0m
status-im.ui.screens.extensions.add.events))
(handlers/register-handler-db
1 file(s) formatted incorrectly
[Pipeline] }
```
igor680
2018-08-31 17:29
Yeah, formatting needs to be fixed. Was it merged already? cc @jeluard
dmitryn
2018-08-31 19:35
i?ve submitted PR with a fix https://github.com/status-im/status-react/pull/5722
george952
2018-09-03 10:34
@jakub @igor680 I wrote a howto:
https://github.com/status-im/status-react/issues/5421#issuecomment-418071329
With any luck we won't need to sync on this and it'll just work, but I'll be available in the afternoon if you want to tackle this today and anything goes wrong that I may be able to help with.
If you'd like me to get the existing certificate+key to you instead of creating a new one, let me know how, GPG, whatever. Thanks!
igor680
2018-09-03 11:04
okay, maybe we will need to pre-setup this osx machines anyway
jakub
2018-09-03 11:44
Why not commit the keychain to the repo and get the key for unlocking it from Jenkins credentials during build?
igor680
2018-09-03 11:50
maybe we should do that
george952
2018-09-03 11:52
That sounds good. Should I go ahead, encrypt the keychain with a 256-bit-entropy alphanumeric password and open a PR?
jakub
2018-09-03 11:54
Yeah, that way we don't have to distribute it to hosts in advance
george952
2018-09-03 11:55
Cool, should I send you the password via Status so you can enter it into Jenkins, mail+GPG, something else?
jakub
2018-09-03 11:56
Gpg sounds good, I can give you my public key after the dev meeting
nastya
2018-09-04 06:07
builds are failing again
igor680
2018-09-04 06:15
Which job is that? Nightlies are green...
nastya
2018-09-04 06:24
igor680
2018-09-04 06:43
no further action needed
context: release builds
reason: https://status-im.slack.com/archives/C801W3L86/p1536042923000100
jeluard
2018-09-04 13:18
@jakub How are those URLs resolved?? https://get.status.im/chat/public/ethberlin
jeluard
2018-09-04 13:19
I'd like to have one for extensions
jakub
2018-09-04 13:19
updated to `master` get published via https://jenkins.status.im/job/misc/job/universal-links-handler/
jeluard
2018-09-04 13:20
Cool! I can add my own JS ?
jakub
2018-09-04 13:21
I'd talk to @andreap
jakub
2018-09-04 13:21
he created it
jeluard
2018-09-04 14:20
@andreap Anything specific I should know? :slightly_smiling_face:
nastya
2018-09-05 11:09
@igor680 curious why do we need desktop builds for each pr in mobile ? maybe we can disable them to save time and build apk and ipa only?
igor680
2018-09-05 11:09
because that way we are sure that we don't break Desktop builds
nastya
2018-09-05 11:09
ah i see
nastya
2018-09-05 11:09
okay sure
igor680
2018-09-05 11:10
and the builds are parallel, so it is not much slower
jakub
2018-09-05 14:04
It is true that the Linux build is the slowest,and hence slows down the whole build by about 4min, maybe we can speed it up so show
nastya
2018-09-05 14:05
eh 4 minutes improvement does not worth it i think )
jakub
2018-09-05 15:12
4 minutes on every single PR build isn't a small improvement, that's almost 25% faster
nastya
2018-09-05 15:20
:okay:
jakub
2018-09-05 15:20
I'll do some investigation, but it's not critical, if there are any cheap wins I'll do it
igor680
2018-09-09 13:15
@jakub it looks like something was done to the mac nodes recently that now everything fails there
see
- https://ci.status.im/job/status-react/job/nightly/673/console
- https://ci.status.im/job/status-react/job/nightly/675/console
they both are building the same commit, but the second one failed. and now we can't build PRs because macos desktop and iOS fail.
igor680
2018-09-09 13:15
most probably something is wrong with node (maybe a wrong version)
jakub
2018-09-09 13:25
Yes, I was playing with that, I guess a node downgrade would be in order
jakub
2018-09-09 13:26
I was going to fix it today in the evening, but if you want to work today I can look into it
jakub
2018-09-09 13:31
hmmm this is tricky
jakub
2018-09-09 13:51
ok, I downgraded it to `8.10.0` with:
```
node icu4c --force --ignore-dependencies
brew install 'https://raw.githubusercontent.com/Homebrew/homebrew-core/faa3cefe0e32950b2587696546e9e0d4afb88886/Formula/node@8.rb' --without-icu4c
```
jakub
2018-09-09 13:51
started another nightly
jakub
2018-09-09 13:51
hopefully this is enough
jakub
2018-09-09 13:52
if not you can try downgrading even further back
jakub
2018-09-09 13:52
I'm going out
jakub
2018-09-09 13:52
back in ~12 h
igor680
2018-09-09 13:52
No, wait
igor680
2018-09-09 13:53
?nvm? is the way to manage node...
igor680
2018-09-09 13:53
we probably should use it and don't bother installing anything by ourselves
jakub
2018-09-09 13:57
This works too
jakub
2018-09-09 13:58
Or, might, we'll see the nightly
jakub
2018-09-09 13:58
Anyway, if you want to use NVM feel free to
jakub
2018-09-09 14:03
I actually checked and didn't see NVM on the macs
igor680
2018-09-09 14:07
yeah, I guess we need to install it on every Mac
jakub
2018-09-09 15:40
```FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory``` this is a weird one
igor680
2018-09-09 15:57
Yeah, on a wrong node version, there is some infinite loop or something
jakub
2018-09-09 17:42
But this is the same version We use on Linux slaves
jakub
2018-09-09 17:42
8.10.0
igor680
2018-09-09 18:46
Idk their versioning confuses me, so I just do `nvm use 9.3.0` and it all works
jakub
2018-09-09 19:17
Hmm, maybe we should have a call to mvn as part of the build process in used in jenkinsfiles
jakub
2018-09-09 19:17
So we are always on the same page
jakub
2018-09-10 01:07
I think the problem might be with npm and node, Id really like to switch to yarn
jakub
2018-09-10 03:44
yup, as far as I can tell the __MAGIC__ version is `npm@5.5.1`, every other version fails regardless of which node version we use, we really should switch to `yarn`, this is ridiculous
jakub
2018-09-10 04:07
but now it's something else...
jakub
2018-09-10 04:07
```# http://github.com/status-im/status-go/vendor/github.com/rjeczalik/notify
../vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:51:216: cannot use nil as type _Ctype_CFAllocatorRef in argument to func literal
../vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:162:47: cannot use nil as type _Ctype_CFAllocatorRef in argument to _Cfunc_CFStringCreateWithCStringNoCopy
../vendor/github.com/rjeczalik/notify/watcher_fsevents_cgo.go:163:225: cannot use nil as type _Ctype_CFAllocatorRef in argument to func literal```
jakub
2018-09-10 04:07
what fresh maddness is this...
jakub
2018-09-10 04:10
oh, right, golang has to be at `1.10`, not `1.11`
jakub
2018-09-10 04:10
```brew switch golang 1.10.3```
jakub
2018-09-10 04:25
now iOS...
jakub
2018-09-10 04:25
```? /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/uri.rb:106:in `require': /usr/local/Cellar/ruby/2.5.1/lib/ruby/2.5.0/uri/generic.rb:212: syntax error, unexpected '.' (SyntaxError)
```
jakub
2018-09-10 04:25
no idea what that is about...
jakub
2018-09-10 04:27
ruby versions or fastlane versions are not controled by brew
jakub
2018-09-10 04:27
so no idea
jakub
2018-09-10 04:27
I'm going to sleep, out tomorrow
jakub
2018-09-10 04:31
okay, it was Ruby version on `macos1`, I bumped it to `2.3.7`
jakub
2018-09-10 05:02
works
jakub
2018-09-10 05:25
geary
pedro
2018-09-10 09:25
at least Jenkins' issues are recognized by the team: https://www.infoq.com/news/2018/09/jenkins-stability-cloud-native
lukasz949
2018-09-10 10:11
Is https://canary.status.im/ working for you? I get 500 after trying to sign in with github account. cc @jakub. My GH is lukaszfryc
chad
2018-09-10 10:13
lol do we need monitoring for canary? I also get a 500
cryptowanderer
2018-09-10 10:15
Who will guard the guards?
dmitry.shulyak
2018-09-10 10:21
something is wrong with http://discuss.status.im. i forgot my password. tried to recover it - no confirmation email. tried to register with another email - same story. @jakub can you check it out please? maybe something needs to be restarted
lukasz949
2018-09-10 11:44
Hey! Looks like we keep builds artifacts (Android, iOS, etc.) only for couple of days. Can we keep them at least for 7 days? I find myself wasting time because I need to rebuild PRs
jakub
2018-09-10 12:31
I'll check
jakub
2018-09-10 12:34
I see no delivery errors for your status email in MailGun
jakub
2018-09-10 12:35
oh
jakub
2018-09-10 12:35
you're using your gmail email
jakub
2018-09-10 12:36
hm, but I see no mailgun errors either
dmitry.shulyak
2018-09-10 12:39
strange. so it says that it sent an email or it just doesn't report any errors?
jakub
2018-09-10 12:40
i seen no emails sent to you
jakub
2018-09-10 12:40
but I do see some older email delivery errors
dmitry.shulyak
2018-09-10 12:40
btw do you see new registrations? i tried to create new account with http://status.im mail
jakub
2018-09-10 12:40
oh yea
jakub
2018-09-10 12:41
```{"type"=>"signup", "user_id"=>121, "email_token"=>"xyz", "to_address"=>"", "current_site_id"=>"default"}```
jakub
2018-09-10 12:41
```Jobs::HandledExceptionWrapper: Wrapped Net::OpenTimeout: execution expired
```
jakub
2018-09-10 12:42
this might be a disucss issue because I see mailgun not mentioning it
jakub
2018-09-10 12:42
can I fix this tomorrow?
jakub
2018-09-10 12:42
I don't want to open that can of worms considering today is my day off
dmitry.shulyak
2018-09-10 12:43
:slightly_smiling_face: definitely, it is not urgent thing
jakub
2018-09-10 12:48
i think it might be the new iptables rules I added
jakub
2018-09-10 12:50
should have worked now
jakub
2018-09-10 12:50
looks like I have to rethink the iptables rules
dmitry.shulyak
2018-09-10 14:02
thanks, mail was delivered
jakub
2018-09-10 14:06
Then bump the limits in the jenkinsfiles
lukasz949
2018-09-10 14:21
@jakub could you point me to a file where I can change it? I looked at jenkinsfiles but I can't find it. I only see `daysToKeepStr` for `logRotator` but I don't think it's the parameter I?m searching for
pedro
2018-09-11 16:01
@jakub seems like Jenkins can't delete a build artifact, so `status-go-manual` can't build: https://ci.status.im/job/status-go/job/status-go-manual/384/console
pedro
2018-09-11 16:01
> Caused by: java.nio.file.FileSystemException: /home/jenkins/workspace/status-go/status-go-manual/src/github.com/status-im/status-go/build/bin/statusgo-android-16.aar: Operation not permitted
jakub
2018-09-11 16:02
looking
jakub
2018-09-11 16:03
cleaned by hand
pedro
2018-09-11 16:04
awesome, thanks!
jakub
2018-09-11 16:04
must have been some remnant of my manual work
jakub
2018-09-11 16:04
because there were some root files
jakub
2018-09-11 16:04
my bad
pedro
2018-09-11 16:04
np
pedro
2018-09-11 16:24
There's something strange going on with `status-go-manual`: if I push to `feature/1180-simple-config`, it triggers a build of `develop` on my name
pedro
2018-09-11 16:25
probably some filter missing somewhere
pedro
2018-09-11 20:04
The mobile iOS build has been flaky the last few days for me, not sure why: https://ci.status.im/job/status-react/job/combined/job/mobile-ios/857/
igor680
2018-09-11 20:11
@jakub looks like ruby version mismatch
jakub
2018-09-11 20:12
is it always `macos01`?
jakub
2018-09-11 20:12
what ruby version do we require?
jakub
2018-09-11 20:12
```
administrators-Mac-Pro:~ administrator$ ruby --version
ruby 2.3.7p456 (2018-03-28 revision 63024) [x86_64-darwin16]
```
jakub
2018-09-11 20:13
```administrators-Mac-Pro:~ administrator$ brew list --versions ruby
ruby 2.5.0_2 2.5.1
```
igor680
2018-09-11 20:20
@jakub ```[33mRun `bundle install` to install missing gems.[0m```
I have 2.3.7 and it works just fine
jakub
2018-09-11 20:27
weird
jakub
2018-09-11 20:29
```Could not find unf_ext-0.0.7.5 in any of the sources (Bundler::GemNotFound)```
jakub
2018-09-11 20:29
it is defined as required in our `Gemfile.lock`
jakub
2018-09-11 20:30
is this supposed to be installed by `bundle`?
jakub
2018-09-11 20:31
we did run `bundle install --quiet` in that job
jakub
2018-09-11 20:31
so what gives?
igor680
2018-09-11 20:32
�\_(?)_/�
jakub
2018-09-11 20:33
i love this shit
jakub
2018-09-11 20:33
I'll look into it, but first I need to buy these freakin' flight tickets
jakub
2018-09-11 20:33
3rd time I'm trying to pay for them
jakub
2018-09-11 20:36
just kill me , please, I don't want to fill out the same forms for the 4th time...
jakub
2018-09-11 20:41
4th time's the charm!
jakub
2018-09-11 21:55
I'll check out the job config
pedro
2018-09-11 22:54
It's zero-indexed, I guess :wink:
jakub
2018-09-11 22:54
HA!
jakub
2018-09-11 22:54
good one
jakub
2018-09-11 22:54
looking into this issue right now
jakub
2018-09-11 22:54
god, i love package managers
jakub
2018-09-11 22:54
the more the merrier
jakub
2018-09-11 22:55
we can never have enough
jakub
2018-09-11 22:55
we should make a few of our own, just to make the world a better place
pedro
2018-09-11 22:55
No rush, I finally got the damn build to pass
pedro
2018-09-11 22:55
Haha!
pedro
2018-09-11 22:56
We could also pulled a few new JavaScript frameworks while we're at it!
pedro
2018-09-11 22:57
There's never too many of those
jakub
2018-09-11 22:57
we're still missing perl modules in `status-react`
jakub
2018-09-11 22:58
we could also add some php as a cherry on top
jakub
2018-09-11 22:58
hmmmm, the branch parameter seems to affect the jenkinsfile
jakub
2018-09-11 22:58
jakub
2018-09-11 22:59
can you show me which build used wrong branch for you?
jakub
2018-09-11 22:59
I see it's `branch` parameter was `feature/1180-simple-config`
jakub
2018-09-11 23:00
```> git rev-parse feature/1180-simple-config^{commit} # timeout=10
> git rev-parse refs/remotes/origin/feature/1180-simple-config^{commit} # timeout=10
Checking out Revision 5cc5e97cf4b732cb3bbee2c4696794bcd17c993d (refs/remotes/origin/feature/1180-simple-config)
> git config core.sparsecheckout # timeout=10
> git checkout -f 5cc5e97cf4b732cb3bbee2c4696794bcd17c993d
Commit message: "fixup! Address PR review comments. Part of #1180"```
jakub
2018-09-11 23:00
looks correct to me
jakub
2018-09-11 23:01
so I'm not sure what you mean
jakub
2018-09-11 23:05
okaaaay...
jakub
2018-09-11 23:06
fastlane > faraday-cookie_jar > http-cookie > domain_name > unf > unf_ext
jakub
2018-09-11 23:06
so there is a rather lenghty dependency on `unf_ext`
jakub
2018-09-11 23:06
which means it SHOULD be installed
jakub
2018-09-11 23:07
```administrators-Mac-Pro:desktop-macos jenkins$ bundle install | grep unf
Using unf_ext 0.0.7.5
Using unf 0.1.4```
jakub
2018-09-11 23:07
I can fucking see it there!
jakub
2018-09-11 23:08
what in the fuck...
jakub
2018-09-11 23:24
after upgrading fastlane to `2.104.0` seems to work fine:
https://ci.status.im/job/status-react/job/combined/job/mobile-ios/863/console
jakub
2018-09-11 23:24
```[00:20:58]: ? Ignoring json-2.1.0 because its extensions are not built. Try: gem pristine json --version 2.1.0
[00:20:58]: ? Ignoring unf_ext-0.0.7.5 because its extensions are not built. Try: gem pristine unf_ext --version 0.0.7.5
[00:20:58]: ? [31mCould not find unf_ext-0.0.7.5 in any of the sources[0m
[00:20:58]: ? [33mRun `bundle install` to install missing gems.[0m```
jakub
2018-09-11 23:25
seems like to build `unf-ext` it needs ruby header files
jakub
2018-09-11 23:27
wait, no fails again
jakub
2018-09-11 23:27
grr
jakub
2018-09-11 23:33
ok, I removed `.gem/ruby/` and I think it works now
jakub
2018-09-11 23:56
dmitryn
2018-09-12 07:42
sergii
2018-09-12 08:25
Hey @jakub Nightly build job fails at start with `groovy.lang.MissingPropertyException: No such property: Jenkinsfile for class: WorkflowScript
at org.codehaus.groovy.runtime.ScriptBytecodeAdapter.unwrap(ScriptBytecodeAdapter.java:53).... ` (https://ci.status.im/job/status-react/job/nightly/686/console)
igor680
2018-09-12 09:11
actually not the same
```
The following build commands failed:
PhaseScriptExecution Run\ Script /Users/jenkins/Library/Developer/Xcode/DerivedData/StatusIm-gzbepwnrlqbukxbmmpfskimspfch/Build/Intermediates.noindex/ArchiveIntermediates/StatusIm/IntermediateBuildFilesPath/ReactNativeConfig.build/Release-iphoneos/ReactNativeConfig.build/Script-EBE4E8281C7BF689000F8875.sh
(1 failure)
[20:43:54]: Exit status: 7
```
igor680
2018-09-12 09:11
RN build failed
dmitryn
2018-09-12 09:19
hm, i see fastlane gem issue again
```
bundler: failed to load command: fastlane (/Users/jenkins/.bundle/bin/fastlane)
NoMethodError: [!] undefined method `each' for nil:NilClass
```
jakub
2018-09-12 11:34
Yeah, forgot to undo a test change
sergii
2018-09-12 14:40
Awesome! I see nightly build has been created, so you fixed it :slightly_smiling_face:
Thanks Jakob!
jakub
2018-09-12 14:42
Apparently Jenkins doesnt like symlinks to jenkinsfiles
jakub
2018-09-12 14:42
I want to rename Jenkinsfile.combined to just Jenkinsfile but to do that painlessly I'd need for a while two identical copies
pedro
2018-09-12 15:03
would be really cool if we had a fixed link for the latest nightly for each platform, something like https://status-im.github.io/nightly/android.
jakub
2018-09-12 21:05
yeah, I saw your script, I'll think about it
jakub
2018-09-12 21:07
the page is hosted on github
jakub
2018-09-12 21:07
maybe something like this could work
pedro
2018-09-13 20:27
Jenkins build agent is again struggling to delete the build folder: https://ci.status.im/job/status-go/job/status-go-manual/400/console
pedro
2018-09-13 20:28
yup
jakub
2018-09-13 20:28
kinda busy with ghost setup right now, but I have this noted =]
jakub
2018-09-13 20:29
it's hard being a one-man army :smile:
pedro
2018-09-13 20:31
no worries, totally understand. Wish we fill that second DevOps opening soon so you can start to breathe a bit!
jakub
2018-09-13 20:31
i think it's a good thing to be working at slightly over capacity
jakub
2018-09-13 20:31
don't want to get complacent :smile:
pedro
2018-09-13 20:32
haha, I think that even with 2 guys we'll have no shortage of work!
jakub
2018-09-13 20:32
true dat
jakub
2018-09-13 20:33
though I kinda enjoy the responsibility of owning everything :smile:
pedro
2018-09-13 20:34
true, and 1 brain is usually easier to coordinate than 2 :wink:
jakub
2018-09-13 20:35
yup, cooperation has it's additional costs
pedro
2018-09-14 09:14
@adamb @igor680 is there any reason to have `status-go-manual` trigger automatically whenever `develop` advances? Right now there's even a bug that causes the job to start no matter which branch is pushed. I'd just disable the GitHub trigger on that job.
adamb
2018-09-14 09:14
In my opinion there is no point. We should only trigger it for new tags
igor680
2018-09-14 09:17
No idea
pedro
2018-09-14 09:18
I'll clone the status-go-manual into a status-go-tag job which instead of being parameterized, will trigger on new tags. How does that sound?
adamb
2018-09-14 09:18
sounds good
adan
2018-09-15 10:49
Hi all, can anybody here provide me ssh access to any mac os x machine in order to set up the infrastructure for running e2e tests against desktop app?
chad
2018-09-15 11:52
@jakub is your man
chad
2018-09-15 11:52
Thanks for running those tests!
jakub
2018-09-15 11:52
already done
chad
2018-09-15 11:53
Thanks!
adan
2018-09-15 12:59
yep, it's solved, thank you @jakub one more time :slightly_smiling_face:
pedro
2018-09-17 15:39
Hey @adamb, due to the https://github.com/status-im/status-react/issues/5786 PR, we need to publish a new status-go version to incorporate in that PR. What is the process for doing that?
adamb
2018-09-17 19:20
Hey. It?s pretty manual but I think it has proven to work well so we can think of making another step and making it automated.
Currently, create a git tag with intended version, push it and to to Releases on GH to create a new one from your tag.
We have v0.14.1 as the last one. I usually bump minor when there are significant changes. In this case, I would go with v0.15.0.
jeluard
2018-09-19 06:18
Does that kind of Jenkins errors rings some bell? `groovy.lang.MissingPropertyException: No such property: cmn for class: WorkflowScript`
https://ci.status.im/blue/organizations/jenkins/status-react%2Fpull%20requests/detail/PR-5834/7/pipeline
jeluard
2018-09-19 07:36
So looks like triggering a new build from the UI is not possible
igor680
2018-09-19 07:37
yeah it looks like *this* UI doesn't work for some reason, only the real jenkins UI works
jeluard
2018-09-19 07:38
Looks better indeed..
jeluard
2018-09-19 07:38
Jenkins is quite creative when it comes to blowing up
igor680
2018-09-19 07:42
yeah
jeluard
2018-09-19 08:42
How about this error that prevents `iOS` builds: `CocoaPods could not find compatible versions for pod "GoogleUtilities/MethodSwizzler":`
jeluard
2018-09-19 08:58
@jakub Please send help! Jenkins doesn't love me anymore!
igor680
2018-09-19 09:22
you probably can do `pod update` locally and commit the update `podfile.lock`
igor680
2018-09-19 09:22
it might help
igor680
2018-09-19 09:23
but cocoapods really suck at being used for teamwork
jeluard
2018-09-19 09:23
I am using linux ..
igor680
2018-09-19 09:24
ookay
you can try adding it to jenkinsfile just before fastlane calls
igor680
2018-09-19 09:24
something like `sh "cd ios && pod update && cd ..` should do
jeluard
2018-09-19 09:25
Can't we have fixed version or something?
jeluard
2018-09-19 09:25
Seems like we would need that for reproducible builds
igor680
2018-09-19 09:25
I've no clue why it does what it does in your PR
igor680
2018-09-19 09:26
we probably should switch to carthage or something like that
jeluard
2018-09-19 09:26
I understand @pedro has the same issue
igor680
2018-09-19 09:26
it just doesn't like linux users :stuck_out_tongue:
jeluard
2018-09-19 09:26
Looks like some dep updates sneaked in
igor680
2018-09-19 09:27
it usually happens if some smart lib developer rewrites history on their github
igor680
2018-09-19 09:34
let me see
igor680
2018-09-19 09:37
hmm, I don't have this error locally :disappointed:
dmitryn
2018-09-19 12:49
I also cannot build 2 PRs because of this error @jakub can you take a look please?
jakub
2018-09-19 12:50
sure, looking now
jakub
2018-09-19 12:52
looks like it's only `macos1`
jakub
2018-09-19 12:52
which is the original macos slave
igor680
2018-09-19 13:00
something wrong with cocoapods versioning?
igor680
2018-09-19 13:00
or it just needs some reset
jakub
2018-09-19 13:01
not sure
jakub
2018-09-19 13:01
removed `/Users/jenkins/.cocoapods` but that did nothing
jakub
2018-09-19 13:01
i'm gonna update cocoapods
igor680
2018-09-19 13:03
1.5.2 is what expected
jakub
2018-09-19 13:05
downgraded from 1.5.3 to 1.5.2 and nothing changed
igor680
2018-09-19 13:09
maybe that will help
jakub
2018-09-19 13:10
nope
jakub
2018-09-19 13:10
same thing
jakub
2018-09-19 13:11
the fuck...
jakub
2018-09-19 13:11
anyway, i took the slave offline so builds should be fine now
igor680
2018-09-19 13:12
what is the full error in our case?
jakub
2018-09-19 13:12
```[!] CocoaPods could not find compatible versions for pod "Firebase/Core":
In Podfile:
Firebase/Core
Specs satisfying the `Firebase/Core` dependency were found, but they required a higher minimum deployment target.
CocoaPods could not find compatible versions for pod "GoogleUtilities/MethodSwizzler":
In Podfile:
Firebase/Core was resolved to 5.8.0, which depends on
FirebaseAnalytics (= 5.1.2) was resolved to 5.1.2, which depends on
GoogleUtilities/MethodSwizzler (~> 5.2.0)
Specs satisfying the `GoogleUtilities/MethodSwizzler (~> 5.2.0)` dependency were found, but they required a higher minimum deployment target.
```
jakub
2018-09-19 13:13
it looks like some dependency hell
jakub
2018-09-19 13:14
but why only on one host...
igor680
2018-09-19 13:15
is there only one cocoapods instance?
igor680
2018-09-19 13:15
just askin...
jakub
2018-09-19 13:15
what do you mean?
jakub
2018-09-19 13:16
is there another `pod` process running?
igor680
2018-09-19 13:16
just maybe "jenkins" user has it's own cocoapods installation for some reason
igor680
2018-09-19 13:16
because it is all weird
jakub
2018-09-19 13:16
no, it doesn't
george952
2018-09-19 14:23
Running into the same issue here
jakub
2018-09-19 14:23
which host?
jakub
2018-09-19 14:23
link to the job?
george952
2018-09-19 14:23
My machine
george952
2018-09-19 14:23
Running make setup
george952
2018-09-19 14:24
And just to make sure, a manual `pod update` in `/ios`
jakub
2018-09-19 14:24
oh, so it is not specific to `macos1`
jakub
2018-09-19 14:24
so maybe the issue is in the cocoapods master repo
jakub
2018-09-19 14:24
seems like it
george952
2018-09-19 14:44
I reset the pods repo with:
```
pod repo remove master
pod setup
```
...and then tried `pod update` again, same thing.
jakub
2018-09-19 14:44
yup tried tthat too
jakub
2018-09-19 14:45
seems like thier repo is broken
george952
2018-09-19 14:46
Recent post with what seems to be the same issue.
george952
2018-09-19 14:46
So yeah, kinda looks like it.
jakub
2018-09-19 14:56
I'm looking at newest issues but I can't find anything matching
george952
2018-09-19 14:57
Oh, me neither, I was suggesting that you create one.
jakub
2018-09-19 14:57
already did
jeluard
2018-09-19 15:06
My PR finally got built so something has been resolved
george952
2018-09-19 15:10
Aaand it just got closed telling us to go to StackOverflow:
https://github.com/CocoaPods/CocoaPods/issues/8110#issuecomment-422837617
jakub
2018-09-19 15:10
Hahahaha
george952
2018-09-19 15:11
It was quick though, gotta give'em that. :slightly_smiling_face:
igor680
2018-09-19 15:12
tbh, it was probably somewhere here https://github.com/CocoaPods/Specs/tree/master/Specs
igor680
2018-09-19 15:12
(the issue)
jakub
2018-09-19 15:13
>Which this doesn't look like.
It's impressive he can tell at just a glance
jakub
2018-09-19 15:14
Well I'm walking to get breakfast, maybe someone can respond ?
george952
2018-09-19 15:15
It's the whack-a-mole school of issue processing.
george952
2018-09-19 15:19
We have a saviour:
https://github.com/CocoaPods/CocoaPods/issues/8110#issuecomment-422842114
george952
2018-09-19 15:19
Kinda. Maybe.
jakub
2018-09-19 15:21
Oh, nice
pedro
2018-09-19 16:25
Hmm, my PR failed again. @jakub, didn't you remove the problematic slave?
jakub
2018-09-19 16:26
it's not just a slave
jakub
2018-09-19 16:26
it seemes there is a bug on cocoapods
pedro
2018-09-19 16:26
ah OK, stopped following the progress on this
pedro
2018-09-19 16:26
txs
jakub
2018-09-19 16:27
oooh
jakub
2018-09-19 16:27
>A workaround is adding `pod 'GoogleUtilities', '5.2.0'` to the Podfile.
jakub
2018-09-19 16:28
we can try that
jakub
2018-09-19 16:28
I'll make a branch
george952
2018-09-19 16:52
I'm getting this now (with the workaround):
```
[!] CocoaPods could not find compatible versions for pod "GoogleUtilities/MethodSwizzler":
In snapshot (Podfile.lock):
GoogleUtilities/MethodSwizzler (= 5.2.2, ~> 5.2.0)
In Podfile:
GoogleUtilities (= 5.2.0) was resolved to 5.2.0, which depends on
GoogleUtilities/MethodSwizzler (= 5.2.0)
Specs satisfying the `GoogleUtilities/MethodSwizzler (= 5.2.2, ~> 5.2.0), GoogleUtilities/MethodSwizzler (= 5.2.0)` dependency were found, but they required a higher minimum deployment target.
```
george952
2018-09-19 16:53
Should we maybe do `GoogleUtilities/MethodSwizzler` in the Podfile?
jakub
2018-09-19 16:53
maybe?
george952
2018-09-19 16:54
Give it a shot, let me know when I can pull
jakub
2018-09-19 16:54
lets wait for the build
george952
2018-09-19 16:55
I'm just trying this locally
jakub
2018-09-19 16:55
oh, then sure, pull
george952
2018-09-19 16:56
OMG it works
jakub
2018-09-19 16:56
holy fucking shit
george952
2018-09-19 16:56
` pod 'GoogleUtilities/MethodSwizzler', '5.2.0'`
george952
2018-09-19 16:56
Is the magic incantation
jakub
2018-09-19 16:56
nice
jakub
2018-09-19 16:56
I'll update the branch
jakub
2018-09-19 16:56
so without ` pod 'GoogleUtilities', '5.2.0'` ?
george952
2018-09-19 16:56
yeah
george952
2018-09-19 16:57
replaced it
jakub
2018-09-19 16:57
motha' fuckin' teamwork
george952
2018-09-19 16:59
Like a bunch of monkeys poking sticks at stuff they don't understand until the gods decide to smile down on them. :smile:
george952
2018-09-19 16:59
...debugging build system failures in a nutshell.
jakub
2018-09-19 16:59
or you can call it that and be a downer
george952
2018-09-19 16:59
hahaha
george952
2018-09-19 16:59
I'll see myself out.
jakub
2018-09-19 17:00
George "Buzzkill" Lipov
george952
2018-09-19 17:00
Yeah, if I ever become a pilot, I'll take that as my call sign.
jakub
2018-09-19 17:01
anyway, the Approve button in https://github.com/status-im/status-react/pull/5905 is feeling a but lonely
jakub
2018-09-19 17:01
give it a poke
george952
2018-09-19 17:02
Poked.
jakub
2018-09-19 17:02
sweet
jakub
2018-09-19 17:02
lets wait for the build and I'll merge
george952
2018-09-19 17:03
Off to the store, bbl
jakub
2018-09-19 17:44
I don't get it
jakub
2018-09-19 17:44
```ArgumentError - Malformed version number string
```
jakub
2018-09-19 17:44
wtf is this
george952
2018-09-19 18:14
Huh
george952
2018-09-19 18:14
The version string is identical to the one I'm using locally and it's working fine here
jakub
2018-09-19 18:14
so this one is something about `macos1` specifically
jakub
2018-09-19 18:15
i merged, I'm gonna investigate `macos1` separately
george952
2018-09-19 18:20
Just confirmed with your branch, works fine here
george952
2018-09-19 20:34
https://github.com/firebase/firebase-ios-sdk/issues/1845#issuecomment-422920516
Should probably remove the workaround and just bump the Firebase version.
roman
2018-09-20 07:05
jenkins is still failing
roman
2018-09-20 07:07
`ArgumentError - Malformed version number string ` during `pod update --silent`
george952
2018-09-20 07:14
@jakub was looking into this yesterday, I think the last status on it was that it only happens on one of the mac hosts. If that's the case - can you try restarting the build in the hope that another mac host will be picked (for now)?
george952
2018-09-20 07:15
It's also related to this whole situation:
https://status-im.slack.com/archives/C8M1Z25J8/p1537346570000100
george952
2018-09-20 07:15
And the workaround for it (see the last replies).
george952
2018-09-20 07:15
The original bug was fixed yesterday, so we can probably remove the workaround and this should be fixed soon :crossed_fingers:
george952
2018-09-20 07:16
(The original issue was with the Firebase dependency)
roman
2018-09-20 07:17
only on macos1?
roman
2018-09-20 07:17
or?
george952
2018-09-20 07:18
That's the last information I got, yeah.
roman
2018-09-20 07:19
i removed macos_pr label from macos1
roman
2018-09-20 07:19
let?s check
roman
2018-09-20 07:19
�\_(?)_/�
george952
2018-09-20 07:20
(it's at the end of the thread I linked btw)
roman
2018-09-20 07:46
so i think it works now
roman
2018-09-20 07:47
temporary solution was to remove `macos_pr` label from `macos1`
tatiana
2018-09-20 08:43
last nightly was failed with same error.
What is solution for now?
tatiana
2018-09-20 08:45
trigger once more?:)
roman
2018-09-20 08:46
idk, nightlies might use another label
tatiana
2018-09-20 08:49
Also in https://ci.status.im/job/status-react/job/combined/job/mobile-ios/1337/console build is failed with same reason
george952
2018-09-20 09:10
Yeah it seems like this is currently only affecting the macos1 host, so if you restart it there's a good chance another host will be picked and it'll work fine.
george952
2018-09-20 09:10
So yes, "have you tried turning it off and on again" :slightly_smiling_face:
jakub
2018-09-20 11:13
you guys, I don't know if you know but there is a "Mark this node temporarily offline" if you know there's a faulty node
jakub
2018-09-20 11:13
just click on the node and take it offline
jakub
2018-09-20 11:13
instead of playing jenkins roulette
jakub
2018-09-20 11:13
that's just silly
pedro
2018-09-20 14:08
Hey @divan, you've been looking into gomobile to replace xgo, right? How's that been going?
I just realized with @jakub that we have regular problems in status-go-manual (e.g. https://ci.status.im/job/status-go/job/status-go-manual/441/console) because the files that are generated by xgo are owned by `root`, since that is the user in the Docker image. I could fork the repo and fix it, but if we're close to transitioning to gomobile, it's probably not worth it
george952
2018-09-20 14:59
I did not. Know that. :facepalming:
divan
2018-09-20 15:17
Hey. There is an open PR with it https://github.com/status-im/status-go/pull/1164
But it should be tested against status-react. I did a couple more attempts to build status-react locally, but it failed.
So it basically waits to be tested.
pedro
2018-09-20 15:34
Thanks, that plays well with what I'm working on (getting Desktop to use status-go artifacts instead of building the library itself through CMake). I may end up testing this
divan
2018-09-20 15:36
That would be awesome!
divan
2018-09-20 15:36
Let me know if/when you're going to test it, so I can jump in and fix things if needed on the fly.
pedro
2018-09-20 15:36
:thumbsup:
pedro
2018-09-20 15:40
I'm giving it a try by forking xgo and fixing the UID issue, but if that ends up causing more problems than it's worth, I'll look into your PR. Right now I'm getting blocked all the time with Jenkins and needing to bother Jakob to clean the repo on the build machine for me
divan
2018-09-20 15:41
xgo is super bloated, I'd rather got rid of it completely
pedro
2018-09-20 15:43
totally, and looking at their repo I could see how challenged they are to fix even a small issue like this:
> Actually, it seems difficult to run the docker container with another user.
Well, they just need to pass `-u $UID` and adapt their script to not use priviledged folders :confused:
pedro
2018-09-20 16:50
OK, managed to fix xgo with an upstream PR, but I do want to get rid of that junk
pedro
2018-09-21 07:36
@divan one thing I noticed is that building iOS now requires XCode, is that expected?
pedro
2018-09-21 07:40
The Jenkins build process would need to be adapted to handle that
divan
2018-09-21 07:42
Hmm... I think this is correct, but let me confirm that
nastya
2018-09-24 09:15
nightly fails https://ci.status.im/job/status-react/job/nightly/708/ with `Error signal (self time 5ms)`
sergii
2018-09-24 11:16
@jakub we had parametrised_build job in the past, but it was archived some time ago. Is there any ability now to create a mobile develop build (apart from re-build nightly for this case)?
jakub
2018-09-24 11:21
Yeah, run a PR build with changed .env.jenkina
sergii
2018-09-24 11:54
Ok, don't really want to create a mess in particular PRs :slightly_smiling_face:
Is it the place you mean need to be modified? https://ci.status.im/job/status-react/job/pull%20requests/view/change-requests/job/PR-5781/configure -> Pipeline tab.
`Definition` = Pipeline script from SCM
jakub
2018-09-24 11:54
what do you want to change?
jakub
2018-09-24 11:55
and please don't touch Pull request build configs
jakub
2018-09-24 11:55
they are quite fragile
sergii
2018-09-24 12:00
> run a PR build with changed .env.jenkina
Where do you mean we need to change env.jenkins settings?
What I need is to create a build same to nightly (so a build from `develop` branch). But using nightly job is not the best way for this because it updates https://status-im.github.io/nightly/ as well.
jakub
2018-09-24 13:16
oh, so you don't want to change any parameters
jakub
2018-09-24 13:16
just want to run a build for a specific commit?
jakub
2018-09-24 13:16
or branch?
jakub
2018-09-24 13:19
https://ci.status.im/job/status-react/job/manual/ - made this, lets see if it works out of the box, or will I have to adjust the Jenkinsfile
sergii
2018-09-24 13:25
Just want to run a build for a specific branch, yeah (`develop` branch).
jakub
2018-09-24 13:25
yeah, #3 is running for `develop`, lets see if it works
noman
2018-09-24 20:00
hey yall, is there a go-to place where you all are deploying docker containers? i'd like to create a docker build for the matrix/whisper bridge since it'll need to run its own ethereum node. i think heroku might be able to run docker containers nowadays? is there somewhere better?
jacqueswww
2018-09-24 20:17
not sure.. but I believe every major cloud service now supports kubernetes/docker.
noman
2018-09-24 20:18
cool, didn't know if anyone had preferences, or if there was already something set up at status to deploy docker thinigs
jacqueswww
2018-09-24 20:18
@jakub could probably allocate something :slightly_smiling_face:
jakub
2018-09-24 20:54
yeah, just write to me
jakub
2018-09-24 20:54
tell me what you want to run and what it needs, I can set it up
noman
2018-09-24 20:54
cool, thank you
tatiana
2018-09-27 07:32
nightly got stuck https://ci.status.im/job/status-react/job/nightly/712/console
Al builds are successful, but job cannot be finished ~4h
@george952 seems that yesterday you made some changes to build process https://github.com/status-im/status-react/pull/5993, can you take a look?
chad
2018-09-27 08:10
Heya @jakub can't seem to get this PR building. Roman has tried restarting a few times with no luck https://github.com/status-im/status-react/pull/6002
yenda
2018-09-27 08:14
can we restart it ?
yenda
2018-09-27 08:16
it needs a rebase
yenda
2018-09-27 08:16
I restarted it
yenda
2018-09-27 08:16
how do we restart a nightly ?
yenda
2018-09-27 08:16
or simply build develop ?
chad
2018-09-27 08:18
:thanks:
andrey
2018-10-03 13:06
```
GitHub has been notified of this commit's build result
Also: groovy.lang.MissingPropertyException: No such property: cmn for class: WorkflowScript
```
andrey
2018-10-03 13:06
@jakub
jakub
2018-10-03 13:07
yeah, if you restart from Blue Ocean for some reason they don't include importedt `*.groovy` scripts that are necessary for the `Jenkinsfile` to work
jakub
2018-10-03 13:08
I think I'll have to open an issue with Jenkins Blue Ocean devs
andrey
2018-10-03 13:28
ok sorry @jakub, please ignore my prev messages
jakub
2018-10-03 13:29
the point is, Jenkinsfiles don't contain all the code to run the builds, that is contained in separate `*.groovy` files that we `load()`
jakub
2018-10-03 13:29
but for some reason when doing restart in Blue Ocean those are not loaded properly
sergii
2018-10-04 12:01
@jakub do you know if anything changed in order to message Slack #jenkins channel with the builds (PR/Nightly/Release?)? I don't see any updates since yesterdays last message https://status-im.slack.com/archives/C8RNGLKRT/p1538562091000100
sergii
2018-10-04 12:04
Yep, something wrong: `ERROR: Slack notification failed. See Jenkins logs for details.` on latest nightly build for slack notification https://ci.status.im/job/status-react/job/nightly/720/console
jakub
2018-10-04 12:33
This is a known issue, if you restart a job from the Blue Ocean UI it will fail this way because it doesn't load the `.groovy` script files
jakub
2018-10-04 12:33
it is a bug in Blue Ocean
jakub
2018-10-04 12:34
I'll check it
jakub
2018-10-04 12:35
```Slack post may have failed. Response: No hooks
Oct 04, 2018 11:11:34 AM WARNING jenkins.plugins.slack.StandardSlackService publish```
jakub
2018-10-04 12:35
oh yeah
jakub
2018-10-04 12:35
someone must have changed something in slack setup
tatiana
2018-10-05 07:11
FYI
sometimes IOS build can be failed with error:
```
org.jenkinsci.plugins.credentialsbinding.impl.CredentialNotFoundException: Could not find credentials entry with ID 'slave-pass-macos3'
at org.jenkinsci.plugins.credentialsbinding.MultiBinding.getCredentials(MultiBinding.java:153)
at org.jenkinsci.plugins.credentialsbinding.impl.StringBinding.bindSingle(StringBinding.java:62)
```
I restarted build couple of times and it helped.
Can you help to fix it?
igor680
2018-10-05 11:15
@jakub should know
andrey
2018-10-05 12:25
`java.io.FileNotFoundException: The file target/ios-prod/cljs/core/async/impl/ioc_helpers.cljs does not exist.`
andrey
2018-10-05 12:25
jakub
2018-10-05 13:52
Yeah, the hosts were renamed slightly, which made w fe builds fail due to gab between that renaming and renaming of credentials that match the hosts
jakub
2018-10-05 13:52
Now it should be fine
dmitryn
2018-10-05 16:02
I don't understand why this MacOs build fails https://ci.status.im/job/status-react/job/combined/job/desktop-macos/2090/console
Locally i can successfully build SnoreNotify
@jakub can i have access to `/Users/jenkins/workspace/status-react/combined/desktop-macos/desktop/modules/react-native-desktop-notification/desktop/SnoreNotify_ep-prefix/src/SnoreNotify_ep-stamp/SnoreNotify_ep-build-*.log` to check the build log?
jakub
2018-10-05 16:04
yeah. thats a pretty useless error
jakub
2018-10-05 16:04
i can add access for you
jakub
2018-10-05 16:04
can you make a PR?
jakub
2018-10-05 16:04
you have to add your key here
jakub
2018-10-05 16:05
and add an entry for your user at the end of this list: https://github.com/status-im/infra-role-bootstrap/blob/master/defaults/main.yml#L85
jakub
2018-10-05 16:05
and increment `uid`
jakub
2018-10-05 16:05
I can merge that PR and deploy your ssh key to all the jenkins hosts
jakub
2018-10-05 16:06
(though in case of macos the users is always `administrator` for ssh)
dmitryn
2018-10-05 16:11
https://github.com/status-im/infra-role-bootstrap/pull/4
My pub key `dmitryn.pub` is already there
jakub
2018-10-05 16:11
oh, cool didn't notice
jakub
2018-10-05 16:11
let me deploy on macos then
jakub
2018-10-05 16:14
```ssh ```
jakub
2018-10-05 16:14
or `02` or `03` depending on which host it ran on
dmitryn
2018-10-05 16:18
thanks a lot, i?m in
jakub
2018-10-05 16:18
cool