Relevant code

Build system requirements

  • Build system needs to support different distributions (and their authentication mechanisms).
    • (Centurion Dan) this needs further discussion - should we support distributions using other git repository systems??
  • A few communities that are important for devuan are waiting on this to happen.
    • (Centurion Dan) Who and what are their requirements and resourc
      • maemo - a name much bigger than devuan, porting the nokia code to non-ancient distros and devuan's been chosen to be the main distro of choice. maemo requires git hosting and CI (jenkins) usage.
      • heads - a project by a devuan developer. requires CI (jenkins) usage and git hosting.
      • DECODE - a project by dyne.org. will probably replicate the infra rather than use devuan's HW
  • Only 4 people in total can issue build requests on gitlab
    • (this is a misconsception as explained in the releasebot section below)
  • GitLab dependency of current setup
  • Current permissions model does not allow for per suite permissions
    • It may be interesting to allow someone to build for ceres/experimental, but not for ascii / jessie. This is not possible.
  • The actual editing of permissions is currently a PITA if it's not just one repo or all repos.
    • Having to manually add a member to 90 repositories –> not fun (this is an issue already)
    • Alternative is to give member all permissions –> not secure (not done)
    • Current result: potential maintainers meet roadblocks often
  • Permissions management is not practical, encourages either bad security practices or being extremely conservative
  • Issuing builds does not scale, opening a GL issue each time is not a good way to solve this.
  • A compromised GL account results in all of Devuan being compromised if that account has amd:msb permissions (all)
    • GL is software, it may have bugs, AFAIK we are not actively updating GL; are we monitoring security notifications? looks like this turns out to be a very critical part of the infrastructure (evilham)
    • IIRC hellekin used to do the updates to gdo. I have no idea if there has been an update since he wandered off many months ago. I have a gitlab account so see the announcements. There have been many of them. Updates often broke things including the custom styling and maybe even the login page. (golinux)
    • updating gitlab is/was a matter of 'apt update && apt upgrade' (Dan has been doing this as required since the migration to our own infra)

Unclear requirements

  • No privilege separation between build issuing and Jenkins job modification
    • (see current_infrastructure/Releasebot/Definitions for permissions nomenclature)
    • MEMBERS GROUP entities have amd:m-b permissions
    • MASTER+ entities on a REPO have —:–b permissions

Solution 1: scorsh

Command authentication and management system (two components):

  • git plugin for the client
  • server that reacts on commit and talks to releasebot

Workflow

* Create a config file for Scorsh

  • Set up WORKERs, which are associated with repository / branch, … (regex involving this two pieces of data)
    • (+) Each WORKER can serve multiple branches / repos / …
  • Each WORKER has a set of COMMANDs defined in its own config file
    • (+) Each COMMAND can be associated with a specific PGP Keyring
    • (+) These COMMANDs can be EXEcution of a script or URL touching
    • (+) The hash of the scripts is saved in the definition
    • (-) No way of telling if config file has changed, utility of hash is limited
  • Use the Scorsh git plugin to create a compatible PGP signed commit
    • (+) The plugin helps create the commit, simplifying interaction
    • (-) Requires additional client software
  • post-commit-hook places a file in a Scorsh spool folder
  • scorshd gets notified about the new file and processes the commit
    • (+) scorshd checks gets the WORKER based on the repository + branch
    • (+) if a WORKER is found, it checks the COMMAND
    • (+) Based on PGP signature and COMMAND description for WORKER it is decided if the COMMAND may run
    • (+) In case of EXE COMMANDs, the script hash is checked
  • Scorsh server passes info to releasebot.
    • (+) “info” are passed through environment variables, set by scorsh and obtained from the commits (suite, distribution, commit number, branch, etc.)
    • (+) the different parts of releasebot would be split into separate commands for “build”, “buildadd”, etc. (see github.com/parazyd/devuan-releasebot)
  • Releasebot passes info to jenkins
  • Build goes on as usual
    • (-) (Centurion Dan) only way is if you make releasebot bypass all it's permissions checks…. this means it's not a minor refactor of releasebot and probably breaks current functionality
  • Uses gpg-signed git commits to trigger pre-configured commands on the server.
  • Each scorsh command (oddly called tags in the beginning) is associated with a whitelist of gpg-keyrings and pre-configured on the server.
  • If commit signed by key in command's gpg whitelist –> exe, else do nothing.
  • Each scorsh command can have several actions. ATM: script exe or touch URL.
    • Script's hash is saved in the configuration and checked before exe.
    • Touch URL –> can be used to trigger jenkins actions.

Criticism

  • Highly customisable
    • (+) Code is well written, easy to understand even if not familiar with go
    • (-) stores project specific data,
    • (-) stores project specific keychain of authorised people
    • (-) May be too complex to solve the issues at hand
    • (-) Lack of accessible example scorsh commit, or PoC (see point below, based on code audit)
      • (-) working at that. I wanted to provide something “self-consistent” and automatic, to avoid the usual critic “it's difficult to setup, so fuck off” (KatolaZ).
    • (-) Has a listener daemon *and* a client
      • The scorsh message is actually YAML at the end of the commit message (*not* the subject) and can easily be added by hand, if the client side is not wanted e.g.:
                Fix bug #123 / Devuanise rsyslog (yay!)            <-- Commit subject
                
                Had to do this and that, decided upon X because Y. <-- Commit body
                Bla bla bla
                
                ---                                                <-- Scorsh msg
                - Cmd: BUILDADD
                  Args:
                    - Arg1
                    - Arg2
                - Cmd: BUILD
                  Args:
                    - Arg1
                    - Arg2
      • (-) also the “client-side” is a silly shell script that inlcudes a scorsh command, but there is no need to have it. It's just easier to have on. Actually, if we want, we can add the scorsh command creation to d1h, and have something like “d1h release” to trigger the remote build, hiding all the complexity (KatolaZ)
    • (-) (Centurion Dan) this also adds permanent cruft to the git commit log comments.
    • (-) The WORKER concept reveals internal implementation details that are not relevant for the actual functioning
      • (KatolaZ) the worker is just a way of separating repo:branch regexp. Each worker is in change of a (possibly list of) repo:branch regexp, which allows granular control on sets of suites/distros
        • also: the structure of workers is totally opaque to the git committer. They just know of a list of scorsh commands that they can use, and nothing else.
      • (?) May be worth revisiting this part of the configuration (see below)
    • (-) A process for adapting permissions needs to be defined, from a talk with parazyd:
      • This could be solved by assigning each distro one (or more) workers and allowing them to edit those via chrooted SFTP
        • (-) (Centurion Dan) this is prone to breakage by misconfiguration - and if ssh is considered a weaker protocol all that pgp protection can be bypassed.
      • Permissions for this by using unix groups
      • Each distro would have one or multiple admins that update this
        • (+) Benefit is: this is scriptable, configuration is text so it's easy to maintain (as opposed to GL issues / membership)
    • (-) If WORKER config is able to define COMMANDs, the fact that those can be any EXE is a concern –> implies full trust in whoever edits that configuration
      • That would be per-distro admins
        • (KatolaZ) scorsh actions (scripts) have associated sha256sums, so if the script does not match the sha256sum it does not get executed
        • (Evilham) BUT! If the Distro-admin can change the WORKER config, they can elaso change the sha256 sum.
        • (KatolaZ) I guess the administration of scorsh might not be delegated to distros. We can configure their commands, but maybe we should not allow much more there? (dunno)
        • (KatolaZ) We could also have a pre-defined list of available commands, though…
      • (+) It can be mitigated by allowing per-distro admins to only edit the keyring but not to define arbitrary COMMANDs (that's a good suggestion, indeed - KatolaZ)
        • this is more of a general scorsh feature/issue as for jenkins we can hardcode the behaviour to the four available commands (build, buildadd, builddel, buildmodify) (~parazyd)
        • Or add config options to the main .cfg, so that WORKERs can't override COMMANDs. Anything is possible, our code (evilham)
        • ^^^ this would be quite easy to implement with a flag (immutable?) in the main scorsh config file
    • Having the permissions stored in a keyring is a bit intransparent, checking who has permissions to what should be very easy and quick.
      • (+) Can easily be done, e.g. with simple bash scripts
      • (+) I also have in mind a related keyblock-manager tool, with a CLI/curses interface, but I haven't talked about it in the list, to avoid further critics (KatolaZ)
        • (Centurion Dan) why not use g
    • (-) (Centurion Dan) You have to unpack the git repo to get commands and determine permissions in addition to gitlab already doing that, and the source build job doing that. That's a non-negligible overhead given the size of some of the repo's

Solution 2: releasebot (improved)

  • GROUP: git.devuan.org/$GROUP/$REPO
  • PERMISSIONS: nomenclature amd:mspb
    • 1st half Add, Modify, Delete: permission on job associated with a repository
    • 2nd half Master, Security, build Production, Build suite: permission to build branches in each repository
      • “Master” refers to “master”
      • “Security” to “-security”
      • “build Production” to “suites/”
      • “Build suite” to “suites/*-proposed” branches
    • Any entity starts with permissions —:—-; permissions are additive on these criteria:
  • MEMBERS GROUP: cfg.get(“jobs”, “members_projectid”) –> /devuan/devuan-repository-masters
    • Being member with master permissions adds amd:m-pb permissions on all repositories
  • SECURITY GROUP: cfg.get(“jobs”, “security_projectid”) –> ?
    • Being member adds —:-s– permissions on all repositories
  • MASTER+ (on a given REPO):
    • Having these privileges adds —:–pb permissions on a given REPO
  • MEMBER REPO:
    • Being memer of the REPO adds —:—b permissions on a given REPO

Workflow

  • Add a plugin to either jenkins or jenkins-debian-glue to validate the uploader that signed the commit or tag (for release versions) is either one of the maintainers or uploaders in the control file, and that they are in the projects web of trust.
  • Commiter authorisation is the concern
    • ??(Centurion Dan) why? if they already have permissions to push to git they probably already have permissions to execute a build com
  • Non-listed developers “can use the devuan-releasebot path to trigger builds”.
    • ??(Centurion Dan) Only GDO admins… if we don't trust them they shouldn't have those privileges … everyone else either has to be at least master on the project or group the project is in.
  • Set the target repo's by patching releasebot to “make it pass the repo type and target for the setting up the repo-job.”
  • “Configuration for repo-jobs is stored in the projects repo-job, and git commit triggered builds can be added by using a pgp web of trust.”
  • Might be best to do a separate sources.xml for each repo type (dak, reprepo, minidak etc) and then configure it either via labels, or setting flags in the issue title or description box
    • another option is wrapping the differences server-side and just running the binaries with the templates, rather than having the raw shell scripts in the XML (~parazyd)
      • (Centurion) … sounds like obfuscation to me and requires minimally jenkins to lookup another server
  • Means for issues to tell us what distribution to add/modify/delete a build project
    • Add it to the issue title parser, e.g. “buildadd {maemo,devuan}” to add a {maemo,devuan} job
  • Need to define package mirrors used for the builds and end result upload.
    • Have one set of jenkins templates and use branch names “{maemo,devuan}/jessie” to derive distribution (and pkg mirrors).
    • Now possible: single git repo, single jenkins job –> support for multiple distributions
  • Job rename: “<distribution|group>_<packagename>-<source|binary|repo>”
  • Another option: setting up a separate jenkins server and releasebot instance for the other distributions, and then filter the issues by group.
    • Not really an option, too short term, PITA to maintain.
    • (Centurion Dan) - it would provide a clear demarcation of project boundaries and simplifies the code changes required to get this running. I could make a patch that supports this approach. But don't ask me to do so unless you agree that this approach will be tried.
  • GitLab dependency:
    • “As long as we have a source of groups and users on single projects it's just matter of adding a library supporting any other systems”
      • depending on any decentralized system for managing permissions brings nothing but problems and having to maintain it every time we switch the git backend (~parazyd)
        • (Centurion Dan) that is a strawman argument - first I'm not advocating decentralized system, but would advocate a federated auth system that provided centralized permissions where it crosses the boundaries between tools, as it does in our git + ci build system.
      • on top of that, changing the git-interface system will require to rewrite some parts of releasebot anyway, since atm al(KatolaZ)
        • (Centurion Dan) But no one is suggesting we shouldn't refactor the gitlab integration to provide a neutral API that can be hooked into other authentication/permissions systems (regardless of whether they are directly integrated as in gitlab or not). It's a matter of priority and resources

Criticism

  • unclear how jenkins-debian-glue needs to be modified in order to support more than just one distribution.
    • (Centurion Dan) In what sense? Anything there must be non distro specific so if we where to say implement gpg validation of signed commits - then the script would need not specify that, but either have that information provided in a standard location or in an environment variable. Also other distro's may need to provide their own jenkins-debian-glue-<distro>-buildenv package.
  • Looking at the commits on git.devuan.org, the Devuan configuration for jenkins-debian-glue is the stuff in the “devuan/” dir.
  • Current state of code requires refactoring and cleanup; same can be achieved with ca. 70% LOC that are more readable
  • If any development is taking place, it's not visible. Not a good approach.
    • (Centurion Dan) that's a frivilous argument that doesn't stand up to scrutiny
  • The way it looks like, this will force other communities to sign up for a GitLab account and have appropriate groups setup. Doesn't seem too doable.
    • (Centurion Dan) Yup, they will have to weigh the pro's and cons vs building their own end to end infrastructure. We can't expend our best resources to support other distro's if that means Devuan doesn't progress. We need to carefully weigh the benefit of other distro's use of our infrastructure to ensure that they bring a net contribution and not a net detriment through distraction and stretching our resources - particularly infrastructure developer and administrator time.
    • If not, it is making assumptions on their infrastructure which may or may not hold (e.g. ability to create “issues” with “labels”).
  • There is no proposal to solve the per-suite privilege issue
    • (Centurion Dan) there is, it's simple, but don't ask me to write the patch right now unless you accept that it will be another if statement in the short term.
    • (KatolaZ) It seems that the whole build infra has been patched along, one patch after another, without an overall planning, tbh.
  • It does not change the current Issue creation workflow, nor the permissions management.
    • (Centurion Dan) the Issue creation workflow works. If you want an on push webhook to trigger builds, that can be done. Evilham has initiated that work for a gitlab webhook. Releasebot would not need changes for this as the webhook receiver would simply run releasebot which would verify the changes.
    • My approach would be to restrict this to pushing signed release tags.
  • In particular proposal to solve the privilege giving issue (takes too long, is clumsy, not scalable)
    • (Centurion Dan) that's an assertion without foundation.
  • I am a bit concerned that the groups/user data is not cryptographically signed, a compromised GitLab account would be disastrous (evilham)
  • (Centurion Dan) as would be a compromised pgp key in scorshes approach
    • setting up gpg keys in a secure manner is hard, and few people bother to do it right. For a starter, who of you has a separate signing sub-key for and has moved their gpg master key to a secure offline system and exported their signing sub keys to their day to day systems. Who has an expiry on their signing subkey??
  • The best protection against this is vigilance and that's why it's important to keep the ci reporting to #devuan-dev. The biggest weakness in any security system is the people that use it. How many have come across stupid passwords like “password” and “secret” and their obvious variants, or the passwords on a note taped to the monitor etc…
  • No definition of the “Project's WoT” –> May very well lead to the very thing scorsh has (per repo/task pgp keyring)
  • (Centurion Dan) False. We grant access to the build systems as our trust grows in those that participate - this is Web of Trust in practise, I will trust Jaromil when he says KatolaZ is a good guy and I should give him access to our servers, but I will still verify through interactions and grant privileges as and when I've become satisfied that KatolaZ understands the implications and responsibilities associated with the privileges I'm bestowing.
  • GPG does not make a web of trust in itself, it only provides mechanisms that make the authentication and validation of the WoT easier. It only takes one trusted but careless or malicious actor to provide what is essentially a rootkit into the WoT. Don't elevate gpg above what it's capable of. It's powerful but can't be expected to prevent against malicious intentions.

Discussion

Summary of the summary (evilham)

This is a short summary of the already short summary on the pad, do read the pad. This is also on the pad. Do edit the pad (this included).

  • 2017-08-12 18:15 UTC
    • This was a rollercoaster, when I started, I was heading to a much different place than where I think I got.
    • My analysis / general comments (feel free to differ, keeping the discussion technical).
  • 2017-08-14 20:18 UTC

Summary

  • The current situation where GitLab is the source of *authorisation* is not really acceptable. There are quite a few reasons ranging from security to it being impractical (details in the pad).
    • (Centurion Dan) I agree gitlab as the source of authorisation is not a good solution but for different reasons. For now it's somewhat acceptable. The question really is, does scorsh in providing something entirely disconnected to gitlab actually solve the problem or just create more ambiguity and thus another vector for attack or accident weakening security further?
  • The current situation where releasebot uses GitLab Issues to decide how to interact with Jenkins is also not even a short-term solution any more.
    • (Centurion Dan) - I disagree with this assessment. Setting up jenkins jobs is a typically one off occurence and I don't think it is a big burden to do it this way - in fact it makes sense as your not having to manually provide any urls etc. Obviously triggering builds is the issue, and that can be solved easily with a webhook and triggers builds on push - uses releasebot unchanged which continues to do it's permission checks as it does now - or whatever additional options we add to releasebot.
A couple things where the proposed options (should) concur
  • The naming convention for the Jenkins jobs:
    • <distribution|group>_<packagename>-<source|binary|repo>
  • The naming convention for the branches
    • “$DISTRO/$release”, e.g. “{maemo,devuan}/jessie”
    • “suites/$release” –> equivalent to “devuan/$release”
  • Valid and authorised PGP signed commits should result in automatic build (creation, modification, triggering, etc.)
    • (Centurion Dan) I don't see PGP giving us anything much here. If you trust a user with permissions to push to a given branch in a project, that is sufficient reason to also allow them to build. PGP validation can only strengthen the assertion that the author of that commit approves the commits leading up to and including that - and it's power is mostly for doing a retrospective review.
      • (KatolaZ) ^^^ The above means that we should give master privileges to all the users that have provided packages to devuan?!?
        • (Centurion_Dan) ^^^ that's a separate issue to do with which permissions are allowed to build what packages.
  • Giving build permissions to a large number of packages in the experimental branch should not be much of a hassle (example use case)
    • (Centurion Dan) this can be included in the same case/check as the -proposed leveraging gitlabs “Developer” privilege group.
  • Checking who has permissions to do what with which packages should be very easy. Right now, this info is hidden behind tons of clicks on GL.
    • Best way to fix this is expose it in a separate tool - use the API to access the info and map it out as a useful report.
About releasebot

Since there is no code to judge, this is gathered from what was said that would be done on Releasebot:

  • There are no real proposed improvements to the current workflow, it keeps GitLab as a source of *authorisation* for Devuan and is extremely vague about the way other distributions provide *authorisation*. It looks like the proposed solution is to make them sign up for GitLab, create dedicated groups for each distribution and have them use the GL Issue workflow. Far from optimal.
    • (Centurion Dan) - that's false. I've laid out multiple times with specifity what's required and how to acheive that, and I will provide the code to solve it along with tests to validate it.
  • Current code is python, but hard to understand python. Refactoring is badly needed, integration tests, etc. Basically a rewrite. Again, in theory this already started, more transparency is needed.
    • (Centurion Dan) I've started with tests suites so we can validate refactoring doesn't break current functionality. As I get functional test coverage I will provide a feature branch with proposed patches for each issue, and where I refactor or add functions I will write unit tests to ensure test coverage for that too.
About scorsh
  • It is go, but it is easy to understand go.
  • Its intent is not replacing Releasebot.
  • Releasebot *could* replace Scorsh in the future though.
  • Is already written, and has a good design, should be easy to modify for the issues that have already been identified (and those that may be identified).
Whatever we do
  • We need a different *authorisation* source, relying on GL for this is insane even short-term.
    • (Centurion Dan) why is it “insane”?
      • because you're relying completely on gitlab issues to build (~parazyd)
        • (Centurion Dan) ^^ that simply doesn't my question. Also the use of the term “insane” is emotive and doesn't describe what the actual problem with relying on gitlab for authorisation is. If we have specific detail we can address them
  • If we consider different options, they may not look much different than Scorsh.
    • (Centurion Dan) We can't consider options until we have a good grasp of the current issues. We can't evaluate scorsh until the issues are defined. I suggest we file specific bugs in the bts for each issue against each component. We can use the namespace 'devuan-infrastructure':
      • for non descript build pipeline issues use 'devuan-infrastructure-build-pipeline'
        • use this for raising issues about integration issues etc
      • for configuration or non bug/wishlist issues with specific components,
        • use the component name eg:
          • devuan-infrastructure-gitlab
          • devuan-infrastructure-jenkins
      • for issues relating to a particular software package eg devuan-releasebot, amprolla:
        • please file bugs, feature requests etc there.
      • Perhaps by using the BTS we can get some clarity.
Middle ground?

Something that has crossed my mind and can be seen as a middle-ground, which may or may not be worth considering:

  • Special permissions repository (per distro) with PGP signed commits by whitelisted keys.
    • /users/EMAIL.{perm,pub} ←- (permissions/info) + public key
    • /groups/DISTRO/GROUP.{perm,members} ←- (perms/info) + members
  • Permissions to be strictly defined in consensus, but must be easy to extend
  • Scorsh-like commands in commit body
  • This would have the benefit of:
    • Being easily adopted by other distributions, using their own infrastructure.
    • Easy (and secure) for us to update devuan's and other distro's permissions.
    • Not at-all depending on GitLab.
    • Giving permissions is scalable.
  • Has the inconvenience that:
    • It would have to be implemented and integrated with releasebot
    • Is not too different from Scorsh in that there is an authorisation source and *some code* reacting to commits
  • On the other hand:
    • Scorsh could be modified to work like this.
    • Releasebot could be rewritten to work like this.