Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
community:development:build_system_improvements [2017/10/04 19:13]
blinkdog Formatting cleanup for releasebot section
community:development:build_system_improvements [2017/10/04 19:33] (current)
blinkdog Final rough formatting
Line 199: Line 199:
     * GPG does not make a web of trust in itself, it only provides mechanisms that make the authentication and validation of the WoT easier. ​ It only takes one trusted but careless or malicious actor to provide what is essentially a rootkit into the WoT.  Don't elevate gpg above what it's capable of.  It's powerful but can't be expected to prevent against malicious intentions.     * GPG does not make a web of trust in itself, it only provides mechanisms that make the authentication and validation of the WoT easier. ​ It only takes one trusted but careless or malicious actor to provide what is essentially a rootkit into the WoT.  Don't elevate gpg above what it's capable of.  It's powerful but can't be expected to prevent against malicious intentions.
  
-===== Raw formatting below ===== +==== Discussion ​====
-<​file>​ +
-Summary of the summary (by evilham) +
-=================================== +
-(2017-08-14 20:18 UTC) +
-Whatever we do, we will need to add commit hooks. +
-I'm not sure https://​github.com/​dyne/​scorsh/​blob/​master/​hooks/​post-receive can be deployed for the whole GitLab instance (or a whole group). +
-Created this Merge Request with a simple Webhook implementation that hopefully would work with either choice.+
  
-(2017-08-12 18:15 UTC)+=== Summary of the summary ​(evilham=== 
 +This is a short summary of the already short summary on the pad, do read the pad. This is also on the pad. Do edit the pad (this included).
  
-This was a rollercoaster,​ when I started, I was heading to a much different place than where I think I got.+  * 2017-08-12 18:15 UTC 
 +    * This was a rollercoaster,​ when I started, I was heading to a much different place than where I think I got
 +    * My analysis / general comments (feel free to differ, keeping the discussion technical).
  
-My analysis / general comments (feel free to differkeeping ​the discussion technical).+  * 2017-08-14 20:18 UTC 
 +    * Whatever we dowe will need to add commit hooks. 
 +    * I'm not sure https://​github.com/​dyne/​scorsh/​blob/​master/​hooks/​post-receive can be deployed for the whole GitLab instance (or a whole group)
 +    * Created this Merge Request with a simple Webhook implementation that hopefully would work with either choice.
  
 +=== Summary ===
 +  * The current situation where GitLab is the source of *authorisation* is not really acceptable. There are quite a few reasons ranging from security to it being impractical (details in the pad).
 +    * (Centurion Dan) I agree gitlab as the source of authorisation is not a good solution but for different reasons. ​ For now it's somewhat acceptable. ​ The question really is, does scorsh in providing something entirely disconnected to gitlab actually solve the problem or just create more ambiguity and thus another vector for attack or accident weakening security further?
  
-This is a short summary of the already short summary on the pad, do read the padThis is also on the pad. Do edit the pad (this included).+  * The current situation where releasebot uses GitLab Issues to decide how to interact with Jenkins ​is also not even a short-term solution any more. 
 +    * (Centurion Dan) - I disagree with this assessment. ​ Setting up jenkins jobs is a typically one off occurence and I don't think it is a big burden to do it this way - in fact it makes sense as your not having to manually provide any urls etc ​Obviously triggering builds ​is the issue, and that can be solved easily with a webhook and triggers builds on push - uses releasebot unchanged which continues to do it's permission checks as it does now - or whatever additional options we add to releasebot.
  
-- The current situation where GitLab is the source of *authorisation* is not really acceptable. There are quite a few reasons ranging from security to it being impractical (details in the pad). +== A couple things where the proposed options (should) concur ​== 
-  _(Centurion Dan) I agree gitlab as the source of authorisation is not a good +  ​* ​The naming convention for the Jenkins jobs: 
-  solution but for different reasons. ​ For now it's somewhat acceptable. ​ The  +    * ''​<​distribution|group>​_<​packagename>​-<​source|binary|repo>​''​ 
-  question really is, does scorsh in providing something entirely disconnected +  ​* ​The naming convention for the branches  
-  to gitlab actually solve the problem or just create more ambiguity and thus +    * ''​"​$DISTRO/​$release"​''​, e.g. ''​"​{maemo,​devuan}/​jessie"​''​ 
-  another vector for attack or accident weakening security further?_ +    * ''​"​suites/​$release"​'' ​--> equivalent to ''​"​devuan/​$release"​''​ 
- +  Valid and authorised ​PGP signed commits should result in automatic build (creation, modification,​ triggering, etc.) 
-- The current situation where releasebot uses GitLab Issues to decide how to interact with Jenkins is also not even a short-term solution any more. +    ​* ​(Centurion Dan) I don't see PGP giving us anything much here.  If you trust a user with permissions to push to a given branch in a project, that is sufficient reason to also allow them to build. ​ PGP validation can only strengthen the assertion that the author of that commit approves the commits leading up to and including that - and it's power is mostly for doing a retrospective review. 
-  _(Centurion Dan) - I disagree with this assessment. ​ Setting up jenkins jobs  +      ​* ​(KatolaZ) ^^^ The above means that we should give master privileges to all the users that have provided packages to devuan?!? 
-  is a typically one off occurence and I don't think it is a big burden to do it +        ​* ​(Centurion_Dan) ^^^ that's a separate issue to do with which permissions are allowed to build what packages. 
-  this way - in fact it makes sense as your not having to manually provide any +  * Giving build permissions to a large number of packages in the experimental branch should not be much of a hassle (example use case) 
-  urls etc.  Obviously triggering builds is the issue, and that can be solved +    ​* ​(Centurion Dan) this can be included in the same case/check as the -proposed leveraging gitlabs "​Developer"​ privilege group. 
-  easily with a webhook and triggers builds on push - uses releasebot unchanged +  ​* ​Checking who has permissions to do what with which packages should be very easy. Right now, this info is hidden behind tons of clicks on GL. 
-  which continues to do it's permission checks as it does now - or whatever +    ​* ​Best way to fix this is expose it in a separate tool - use the API to access the info and map it out as a useful report.
-  additional options we add to releasebot._ +
- +
-A couple things where the proposed options (should) concur: +
-The naming convention for the Jenkins jobs: +
-  "​<​distribution|group>​_<​packagename>​-<​source|binary|repo>​" +
-The naming convention for the branches +
-  "​$DISTRO/​$release",​ e.g. "​{maemo,​devuan}/​jessie"​ +
-  "​suites/​$release"​ --> equivalent to "​devuan/​$release"​ +
-- Valid(*PGP signed commits should result in automatic build (creation, modification,​ triggering, etc.) +
-  _(Centurion Dan) I don't see PGP giving us anything much here.  If you trust  +
-  ​a user with permissions to push to a given branch in a project, that is  +
-  ​sufficient reason to also allow them to build. ​ PGP validation can only  +
-  ​strengthen the assertion that the author of that commit approves the commits +
-  ​leading up to and including that - and it's power is mostly for doing a  +
-  ​retrospective review._ +
-  (KatolaZ) ^^^ The above means that we should give master privileges to all the users that have  +
-     provided packages to devuan?!?  +
-  (Centurion_Dan) ^^^ that's a separate issue to do with which permissions are +
-  ​allowed to build what packages. +
-      +
-  ​(*) Valid here means both valid and authorised +
-Giving build permissions to a large number of packages in the experimental branch should not be much of a hassle (example use case) +
-  _(Centurion Dan) this can be included in the same case/check as the -proposed +
-  ​leveraging gitlabs "​Developer"​ privilege group._ +
-Checking who has permissions to do what with which packages should be very easy. Right now, this info is hidden behind tons of clicks on GL. +
-  Best way to fix this is expose it in a separate tool - use the API to access the info and map it out as a useful report.+
  
 +== About releasebot ==
 Since there is no code to judge, this is gathered from what was said that would be done on Releasebot: Since there is no code to judge, this is gathered from what was said that would be done on Releasebot:
-There are no real proposed improvements to the current workflow, it keeps GitLab as a source of *authorisation* for Devuan and is extremely vague about the way other distributions provide *authorisation*. It looks like the proposed solution is to make them sign up for GitLab, create dedicated groups for each distribution and have them use the GL Issue workflow. Far from optimal. +  * There are no real proposed improvements to the current workflow, it keeps GitLab as a source of *authorisation* for Devuan and is extremely vague about the way other distributions provide *authorisation*. It looks like the proposed solution is to make them sign up for GitLab, create dedicated groups for each distribution and have them use the GL Issue workflow. Far from optimal. 
-  (Centurion Dan) - that's false. ​ I've laid out multiple times with specifity what's required and how to acheive that, and I will provide the code to solve it +    ​* ​(Centurion Dan) - that's false. ​ I've laid out multiple times with specifity what's required and how to acheive that, and I will provide the code to solve it along with tests to validate it. 
-along with tests to validate it. +  ​* ​Current code is python, but hard to understand python. Refactoring is badly needed, integration tests, etc. Basically a rewrite. Again, in theory this already started, more transparency is needed. 
-Current code is python, but hard to understand python. Refactoring is badly needed, integration tests, etc. Basically a rewrite. Again, in theory this already started, more transparency is needed. +    ​* ​(Centurion Dan) I've started with tests suites so we can validate refactoring doesn'​t break current functionality. As I get functional test coverage I will provide a feature branch with proposed patches for each issue, and where I refactor or add functions I will write unit tests to ensure test coverage for that too.
- ​(Centurion Dan) I've started with tests suites so we can validate refactoring doesn'​t break current functionality. ​ As I get functional test coverage I will provide a feature branch with proposed patches for each issue, and where I refactor or add functions I will write unit tests to ensure test coverage for that too.+
  
 +== About scorsh ==
 +  * It is go, but it is easy to understand go.
 +  * Its intent is not replacing Releasebot.
 +  * Releasebot *could* replace Scorsh in the future though.
 +  * Is already written, and has a good design, should be easy to modify for the issues that have already been identified (and those that may be identified).
  
-About Scorsh: +== Whatever we do == 
-It is go, but it is easy to understand go. +  * We need a different *authorisation* source, relying on GL for this is insane even short-term. 
-- Its intent ​is not replacing Releasebot+    * (Centurion Dan) why is it "​insane"?​ 
-- Releasebot ​*could* replace ​Scorsh in the future though+      * because you're relying completely on gitlab issues ​to build (~parazyd) 
-Is already writtenand has good designshould be easy to modify for the issues that have already been identified (and those that may be identified).+        * (Centurion Dan) ^^ that simply doesn'​t my question. ​ Also the use of the term "​insane"​ is emotive and doesn'​t describe what the actual problem with relying on gitlab for authorisation ​is.  If we have specific detail we can address them 
 +  If we consider different options, they may not look much different than Scorsh
 +    * (Centurion Dan) We can't consider options until we have a good grasp of the current issues. ​ We can't evaluate scorsh until the issues are defined. ​ I suggest we file specific bugs in the bts for each issue against each component We can use the namespace '​devuan-infrastructure': ​ 
 +      * for non descript build pipeline issues use '​devuan-infrastructure-build-pipeline'​ 
 +        * use this for raising issues about integration issues etc 
 +      * for configuration or non bug/​wishlist issues with specific components 
 +        * use the component name eg: 
 +            * ''​devuan-infrastructure-gitlab''​ 
 +            * ''​devuan-infrastructure-jenkins''​ 
 +      * for issues relating to particular software package eg devuan-releasebotamprolla: 
 +        * please file bugs, feature requests etc there. 
 +      * Perhaps by using the BTS we can get some clarity.
  
-Whatever we do: +== Middle ground== 
-- We need a different *authorisation* source, relying on GL for this is insane even short-term. +Something ​that has crossed ​my mind and can be seen as a middle-groundwhich may or may not be worth considering:​ 
-  (Centurion Dan) why is it "​insane"​+  ​* Special permissions repository ​(per distrowith PGP signed commits by whitelisted keys. 
-  - because you're relying completely on gitlab issues to build (~parazyd) +      * ''/​users/​EMAIL.{perm,pub}'' ​<-- (permissions/​info) + public key 
-    (Centurion Dan) ^^ that simply doesn'​t ​my question. ​ Also the use of the term "​insane"​ is emotive ​and doesn'​t describe what the actual problem with relying on gitlab for authorisation is.  If we have specific detail we can address them +      ''/​groups/DISTRO/​GROUP.{perm,members}''​ <-- (perms/​info) ​members 
-If we consider different optionsthey may not look much different than Scorsh. +  * Permissions ​to be strictly defined in consensusbut must be easy to extend 
-  (Centurion DanWe can't consider options until we have a good grasp of the current issues We can't evaluate scorsh until the issues are defined I suggest we file specific bugs in the bts for each issue against each component. ​ We can use the namespace ​'devuan-infrastructure':  +  * Scorsh-like commands in commit body
-   + for non descript build pipeline issues use '​devuan-infrastructure-build-pipeline'​ +
-     ​use this for raising issues about integration issues etc +
-   + for configuration or non bug/wishlist issues with specific components +
-    use the component name eg: +
-       ​`devuan-infrastructure-gitlab` +
-       ​`devuan-infrastructure-jenkins` +
-   for issues relating ​to a particular software package eg devuan-releasebotamprolla: +
-     please file bugs, feature requests etc there. +
-   ​Perhaps by using the BTS we can get some clarity.+
  
-Something that has crossed my mind and can be seen as a middle-ground,​ which may or may not be worth considering+  * This would have the benefit of
-Special permissions repository (per distro) with PGP signed commits ​by whitelisted keys+    Being easily adopted ​by other distributions,​ using their own infrastructure
-    ​- /​users/​EMAIL.{perm,​pub} <​-- ​(permissions/info) + public key +    ​* Easy (and secure) for us to update devuan'​s and other distro'​s ​permissions. 
-    - /​groups/​DISTRO/​GROUP.{perm,​members} <-- (perms/​info) + members +    ​* Not at-all depending on GitLab
-Permissions to be strictly defined in consensus, but must be easy to extend +    Giving permissions is scalable.
-* Scorsh-like commands in commit body+
  
-- This would have the benefit of+  * Has the inconvenience that
-  - Being easily adopted by other distributions,​ using their own infrastructure. +    * It would have to be implemented ​and integrated with releasebot 
-  - Easy (and secure) for us to update devuan'​s ​and other distro'​s permissions. +    * Is not too different from Scorsh in that there is an authorisation source and *some code* reacting to commits
-  - Not at-all depending on GitLab. +
-  - Giving permissions ​is scalable.+
  
-- Has the inconvenience that+  * On the other hand
-  - It would have to be implemented and integrated with releasebot +    * Scorsh could be modified to work like this. 
-  - Is not too different from Scorsh in that there is an authorisation source and *some code* reacting ​to commits+    Releasebot could be rewritten ​to work like this.
  
-- On the other hand: 
-  - Scorsh could be modified to work like this. 
-  - Releasebot could be rewritten to work like this. 
-</​file>​