From 5b0e72082552dd648b5349d3c42cdf020c4cf479 Mon Sep 17 00:00:00 2001 From: Bhushan Khanale Date: Thu, 16 May 2019 11:54:57 +0530 Subject: [PATCH 1/2] cEP-0031: Improve Generic Bear Quality Closes https://github.com/coala/cEPs/issues/183 --- README.md | 1 + cEP-0031.md | 320 ++++++++++++++++++++++++++++++++++++++++++++++++++++ 2 files changed, 321 insertions(+) create mode 100644 cEP-0031.md diff --git a/README.md b/README.md index bed025fea..767396fc9 100644 --- a/README.md +++ b/README.md @@ -27,4 +27,5 @@ in the `cEP-0000.md` document. | [cEP-0027](cEP-0027.md) | coala Bears Testing API | This cEP describes the implementation process of `BaseTestHelper` class and `GlobalBearTestHelper` class to improve testing API of coala as a part of the [GSoC'18 project](https://summerofcode.withgoogle.com/projects/#6625036551585792). | | [cEP-0029](cEP-0029.md) | Support TOML as a configuration format | This cEP describes the implementation process how support for TOML will be implemented and integrated with the existing configuration system as a part of the [GSoC'19 project](https://summerofcode.withgoogle.com/projects/#6388671438127104). | | [cEP-0030](cEP-0030.md) | Next Generation Action System | This cEP describes the details about Next Generation Action System which will allow bears to define their own actions as a part of [GSoC'19 project](https://summerofcode.withgoogle.com/projects/#5450946933424128). | +| [cEP-0031](cEP-0031.md) | Improve Generic Bear Quality | This cEP describes the improvement in the generic bears currently available along with the implementation of the new bears as a part of the [GSoC'19 project](https://summerofcode.withgoogle.com/projects/#4866569388163072). | | [cEP-0036](cEP-0036.md) | Gitmate for coala | This cEP describes the improvements to IGitt and Gitmate, which include upgrading dependencies, making it functional, changes to plugins, etc. as a part of the [GSoC'21 project](https://summerofcode.withgoogle.com/projects/#6263057774280704). | diff --git a/cEP-0031.md b/cEP-0031.md new file mode 100644 index 000000000..699601511 --- /dev/null +++ b/cEP-0031.md @@ -0,0 +1,320 @@ +# Improve Generic Bear Quality + +| Metadata | | +| -------- | ------------------------------------------- | +| cEP | 31 | +| Version | 0.1 | +| Title | Improve Generic Bear Quality | +| Authors | Bhushan Khanale | +| Status | Proposed | +| Type | Feature | + +## Abstract + +This cEP describes the improvement in the generic bears currently available +along with the implementation of the new bears as a part of the +[GSoC 2019 Project][project]. + +## Introduction + +coala has few generic bears which have the potential to perform better +fixing some of the issues in them. Some of these issues are as follows: + +- [coala/coala-bears#644][644]: Ignore doc comments/doc strings + Comments or docstrings do not have any set standards while the + IndentationBear does not ignore them in a file. IndentationBear checks for + correct indentation in the program statements since that is crucial for + proper code style. Docstrings/comments, on the other hand, do not have any + such compulsion. So ideally docstrings should be ignored by the bear. coala + already uses documentation extraction algorithms in DocumentationStyleBear. + Using the existing algorithms and constructing ignore ranges out of the + extracted comments inside the IndentationBear, this issue can be solved. + +- [coala/coala-bears#1897][1897]: Show PEP8 error description + On running PEP8Bear the default message is 'The code does not comply to + PEP8.' which is not very helpful for the user. This issue can be solved + by improving autopep8 itself. One of the alternative approach is to + invoke pycodestyle inside PEP8Bear and pass the issues to the autopep8 + and generate a fix for them using `--line-range`, `--column-range` and + `--select` options. + +- [hhatto/autopep8#227][227]: Create diff for a specific pep8 issue + User should have a choice of selecting the issue he wants to generate a fix + for. This issue is related to the above issue since the fix for this issue + will introduce a new setting `--column-range` in autopep8. + +Along with fixing these issues, new bears including `OutdatedDependencyBear`, +`RegexLintBear`, `FileModeBear` and `RequirementsCheckBear` +will be implemented. + +In place of `FilesExistBear` we will introduce two new settings in coala, +`require_files_not_empty` and `require_files_for_each_glob` which will check +`files` option doesn't evaluate to empty and there is at least one match for +each glob in `files`. + +## Implementation of new bears + +### 1. OutdatedDependencyBear + +Issue: [coala/coala-bears#2445][2445] + +The bear will be looking for the outdated dependencies in the project. The +dependencies can be specified explicitly with the files or by default the bear +will look for files like requirements.txt, package.json, etc. + +The bear will require changes to the existing [package_manager][package_manager] +with the help of new additions to the requirement classes. + +- For pip requirements, we could use the PyPI JSON API which provides the + version info by specifying the `package_name`. + +```py +from xmlrpc import client + +class PipRequirement(PackageRequirement): + def get_latest_version(self): + """ + Gets the latest version available for the package. + + :return: + A string with the latest version of the package. + """ + pypi = client.ServerProxy(PYPI_URL) + return pypi.package_releases(self.package)[0] +``` + +- For npm dependencies, we can use the npm cli command `npm outdated` which + produces an output as below: + +``` +$ npm outdated +Package Current Wanted Latest Location +glob 5.0.15 5.0.15 6.0.1 test-outdated-output +nothingness 0.0.3 git git test-outdated-output +npm 3.5.1 3.5.2 3.5.1 test-outdated-output +local-dev 0.0.3 linked linked test-outdated-output +once 1.3.2 1.3.3 1.3.3 test-outdated-output +``` + +The output can be directly parsed and can be used to get the list of all +outdated dependencies. + +For pip, the bear can be constructed as below: + +```py +class OutdatedDependencyBear(LocalBear): + def run(self, filename, file, requirement_type: str,): + """ + Checks for the outdated dependencies in a project. + :param requirement_type: + One of the requirement types supported by coala's package manager. + :param requirements_file: + Requirements file can be specified to look for the requirements. + """ + requirement_types = ['pip'] + + if requirement_type not in requirement_types: + raise ValueError('Currently the bear only supports {} as ' + 'requirement_type.' + .format(', '.join( + _type for _type in requirement_types))) + + message = ('The requirement {} with version {} is not ' + 'pinned to its latest version {}.') + + out = run('pip-compile -n --allow-unsafe {}'.format(filename), + stdout=Capture()) + + data = [line for line in out.stdout.text.splitlines() + if '#' not in line and line] + + for requiremenent in data: + package, version = requiremenent.split('==') + pip_requirement = PipRequirement(package) + latest_ver = pip_requirement.get_latest_version() + line_number = [num for num, line in enumerate(file, 1) + if package in line.lower()] + + if LooseVersion(version) < LooseVersion(latest_ver): + yield Result.from_values(origin=self, + message=message.format( + package, + version, + latest_ver), + file=filename, + line=line_number[0], + end_line=line_number[0], + ) +``` + +### 2. RegexLintBear + +Issue: [coala/coala-bears#1532][1532] + +The task of the bear is to check the regex in strings. This can be done using +the AnnotationBear to detect all the regex strings and then check for the valid +regex through the prepared algorithm for each type. + +For Python, the bear can be written as follows: + +```py +class RegexLintBear(LocalBear): + def run(self, filename, file, language: str): + """ + Bear for linting regex through regexlint. + :param language: + The programming language of the file(s). + """ + section = Section('') + section.append(Setting('language', language)) + bear = AnnotationBear(section, Queue()) + + with execute_bear(bear, filename, file) as result: + for src_range in result[0].contents['strings']: + src_line = src_range.affected_source({filename: file})[0] + regex = src_line[src_range.start.column:src_range.end.column-1] + with suppress(re.error): + re.compile(regex) + out = run('regexlint --regex "{}"'.format(regex), + stdout=Capture()).stdout.text + if out[-3:-1] != 'OK': + yield Result.from_values( + origin=self, + message=out, + file=filename, + line=src_range.start.line, + column=src_range.start.column, + end_line=src_range.end.line, + end_column=src_range.end.column, + ) +``` + +### 3. FileModeBear + +Issue: [coala/coala-bears#2370][2370] + +The bear will check the permissions on the files provided by the user +and to ensure that the file permissions are the one that is expected. + +The bear will be used as follows: + +``` +[all.mode] +bears = FileModeBear +filemode = rw + +[all.shell.mode] +bears = FileModeBear +filemode = rwx +``` + +The bear will first find the permissions of the files specified by the user and +then if the permissions are not suitable, the bear will try to change those +permissions into the expected ones. If the bear doesn't have enough permissions +to do so then the bear will let the user know about this. + +```py +class FileModeBear(LocalBear): + def run(self, + filename, + file, + filemode: str, + ): + """ + The bear will check if the file has required permissions provided by + the user. + :param filemode: + Filemode to check, e.g. `rw`, `rwx`, etc. + """ + st = os.stat(filename) + permissions = {'r': stat.S_IRUSR, + 'w': stat.S_IWUSR, + 'x': stat.S_IXUSR, + } + + invalid_chars = [ch for ch in filemode if ch not in permissions] + + if invalid_chars: + raise ValueError('Unable to recognize character `{}` in filemode ' + '`{}`.'.format(''.join(invalid_chars), filemode)) + + mode = st.st_mode + for char in filemode: + if not mode & permissions[char]: + message = ('The file permissions are not adequate. ' + 'The permissions are set to {}' + .format(stat.filemode(mode))) + return [Result.from_values(origin=self, + message=message, + severity=RESULT_SEVERITY.INFO, + file=filename)] +``` + +### 4. RequirementsCheckBear + +Issue: [coala/coala-bears#1113][1113] + +The bear will be focused on Python only since they are most prone for +conflicting requirements. + +The implementation is based on the recursive check for the requirements of each +package in requirements.txt file of the project. The requirements are checked +through PyPI's JSON API. + +```py +class RequirementsCheckBear(GlobalBear): + """ + The bear to check and find any conflicting pip dependencies. + """ + def run(self, require_files: tuple): + """ + :param require_files: + Tuple of requirements files. + """ + data = '' + orig_file = '' + + for require_file in require_files: + if not os.path.isfile(os.path.abspath(require_file)): + raise ValueError('The file \'{}\' doesn\'t exist.' + .format(require_file)) + + with open(require_file) as _file: + content = _file.read() + if not orig_file: + orig_file = content + else: + data += content + + with open(require_files[0], 'a+') as temp_file: + temp_file.write(data) + + out = capture_both('pip-compile {} -r -n --no-annotate --no-header ' + '--no-index --allow-unsafe'.format(require_files[0])) + + if out.stderr.text and not out.stdout.text: + pip_warning = 'Cache entry deserialization failed, entry ignored' + lines = out.stderr.text.splitlines() + lines = [line for line in lines if line not in pip_warning] + yield Result(self, + message=lines[0], + severity=RESULT_SEVERITY.MAJOR, + ) + + with open(require_files[0], 'w+') as _file: + _file.write(orig_file) +``` + +[project]: https://summerofcode.withgoogle.com/projects/#4866569388163072 + +[227]: https://github.com/hhatto/autopep8/issues/227 + +[644]: https://github.com/coala/coala-bears/issues/644 + +[1113]: https://github.com/coala/coala-bears/issues/1113 + +[1532]: https://github.com/coala/coala-bears/issues/1532 + +[1897]: https://github.com/coala/coala-bears/issues/1897 + +[2445]: https://github.com/coala/coala-bears/issues/2445 From 4e20d96f732ea23dd56db69bdc2f27a425ea2d1d Mon Sep 17 00:00:00 2001 From: Abhinav Kaushlya Date: Sun, 6 Jun 2021 11:55:16 +0530 Subject: [PATCH 2/2] Fix remark issues for recent changes --- cEP-0029.md | 2 +- cEP-0030.md | 36 ++++++++++++------------- cEP-0031.md | 48 ++++++++++++++++----------------- cEP-0036.md | 76 ++++++++++++++++++++++++++++++----------------------- 4 files changed, 86 insertions(+), 76 deletions(-) diff --git a/cEP-0029.md b/cEP-0029.md index 8cbefbe6c..3f71df426 100644 --- a/cEP-0029.md +++ b/cEP-0029.md @@ -299,10 +299,10 @@ $ coala --create-coafile .coafile.toml The loaded sections from either .coafile or .coafile.toml can be supplied to: + - TomlConfWriter to create TOML config file - ConfWriter to create coafile - ### Support coala-quickstart to generate config files in TOML. coala-quickstart will be made to generate config files in TOML. diff --git a/cEP-0030.md b/cEP-0030.md index ec12a949a..bd465cb04 100644 --- a/cEP-0030.md +++ b/cEP-0030.md @@ -12,18 +12,18 @@ # Abstract This cEP describes the details about Next Generation Action System which -will allow bears to define their own actions as a part of +will allow bears to define their own actions as a part of [GSoC'19 project](https://summerofcode.withgoogle.com/projects/#5450946933424128). # Introduction -Bears run some analysis on a piece of code and output is in the form of -a `Result` object. Then some action from a predefined set of actions -is applied to that `Result` object. This system is a bit restrictive -as only an action from predefined set of actions can be taken. -If there is a system which will support bears defining their +Bears run some analysis on a piece of code and output is in the form of +a `Result` object. Then some action from a predefined set of actions +is applied to that `Result` object. This system is a bit restrictive +as only an action from predefined set of actions can be taken. +If there is a system which will support bears defining their own actions, then it will make bears more useful. -This project is about modifying the current action system so that bears +This project is about modifying the current action system so that bears can define their own actions. This project also aims to implement a new action which will help in supporting for bears to provide multiple patches for the affected code. @@ -63,7 +63,7 @@ class Result: applied_actions: dict = {}, actions: list = []): - # A new attribute `actions` is added + # A new attribute `actions` is added self.actions = actions # A new parameter `actions` is added @@ -102,7 +102,7 @@ class Result: ## Modifying `ConsoleInteraction` module 1. `ConsoleInteraction` module needs to be modified to add the actions in -`result.actions` to the list of predefined actions when asking user +`result.actions` to the list of predefined actions when asking user for an action to apply 2. For this `acquire_actions_and_apply` function needs to be modified. @@ -156,7 +156,7 @@ def autoapply_actions(results, file_diff_dict, section, log_printer=None): - + bear_actions = [] # bear defined actions from all the results are added to `bear_actions`. for result in results: @@ -188,7 +188,7 @@ def autoapply_actions(results, # This condition checks that if action is in bear_actions which means # that default action is one defined by a bear, then action must be in - # result.actions because then only that action can be applied to that + # result.actions because then only that action can be applied to that # result. if action not in bear_actions or action in result.actions: applicable = action.is_applicable(result, @@ -243,7 +243,7 @@ def get_default_actions(section, bear_actions): # `action_dict` now contains all the actions from `ACTIONS` as well as # bear_actions. - # bears_actions contain action objects, to be consistent with this + # bears_actions contain action objects, to be consistent with this # `ACTIONS` was changed to contain action objects. action_dict = {action.get_metadata().name: action for action in ACTIONS + bear_actions} @@ -264,13 +264,13 @@ def get_default_actions(section, bear_actions): ``` 4. Auto applying actions specific to bears is same as auto-applying -predefined actions. Users just need to add +predefined actions. Users just need to add `default_actions = SomeBear: SomeBearAction` in coafile to autoapply `SomeBearAction` on a result whose origin is `SomeBear`. ## Writing bear specific actions -1. The above changes will now allow bears to define their own actions and +1. The above changes will now allow bears to define their own actions and user can apply these actions interactively or by default. 2. While writing any bear specific actions user must implement `is_applicable` and `apply` method with correct logic. User can also add a @@ -321,9 +321,9 @@ class EditCommitMessageAction(ResultAction): ### AddNewlineAction for GitCommitBear -1. `AddNewlineAction` is an action specific to `GitCommitBear`. Whenever -`GitCommitBear` detects that there is no newline between shortlog and -body of the commit message it will yield a `Result` and pass +1. `AddNewlineAction` is an action specific to `GitCommitBear`. Whenever +`GitCommitBear` detects that there is no newline between shortlog and +body of the commit message it will yield a `Result` and pass `AddNewlineAction` as an argument. ```python @@ -413,7 +413,7 @@ class Result: actions: list = [], alternate_diffs: (list,None) = None): - # A new attribute `alternate_diffs` is added + # A new attribute `alternate_diffs` is added self.alternate_diffs = alternate_diffs # A new parameter `alternate_diffs` is added diff --git a/cEP-0031.md b/cEP-0031.md index 699601511..c79f7e3a5 100644 --- a/cEP-0031.md +++ b/cEP-0031.md @@ -20,28 +20,28 @@ along with the implementation of the new bears as a part of the coala has few generic bears which have the potential to perform better fixing some of the issues in them. Some of these issues are as follows: -- [coala/coala-bears#644][644]: Ignore doc comments/doc strings - Comments or docstrings do not have any set standards while the - IndentationBear does not ignore them in a file. IndentationBear checks for - correct indentation in the program statements since that is crucial for - proper code style. Docstrings/comments, on the other hand, do not have any - such compulsion. So ideally docstrings should be ignored by the bear. coala - already uses documentation extraction algorithms in DocumentationStyleBear. - Using the existing algorithms and constructing ignore ranges out of the - extracted comments inside the IndentationBear, this issue can be solved. - -- [coala/coala-bears#1897][1897]: Show PEP8 error description - On running PEP8Bear the default message is 'The code does not comply to - PEP8.' which is not very helpful for the user. This issue can be solved - by improving autopep8 itself. One of the alternative approach is to - invoke pycodestyle inside PEP8Bear and pass the issues to the autopep8 - and generate a fix for them using `--line-range`, `--column-range` and - `--select` options. - -- [hhatto/autopep8#227][227]: Create diff for a specific pep8 issue - User should have a choice of selecting the issue he wants to generate a fix - for. This issue is related to the above issue since the fix for this issue - will introduce a new setting `--column-range` in autopep8. +- [coala/coala-bears#644][644]: Ignore doc comments/doc strings + Comments or docstrings do not have any set standards while the + IndentationBear does not ignore them in a file. IndentationBear checks for + correct indentation in the program statements since that is crucial for + proper code style. Docstrings/comments, on the other hand, do not have any + such compulsion. So ideally docstrings should be ignored by the bear. coala + already uses documentation extraction algorithms in DocumentationStyleBear. + Using the existing algorithms and constructing ignore ranges out of the + extracted comments inside the IndentationBear, this issue can be solved. + +- [coala/coala-bears#1897][1897]: Show PEP8 error description + On running PEP8Bear the default message is 'The code does not comply to + PEP8.' which is not very helpful for the user. This issue can be solved + by improving autopep8 itself. One of the alternative approach is to + invoke pycodestyle inside PEP8Bear and pass the issues to the autopep8 + and generate a fix for them using `--line-range`, `--column-range` and + `--select` options. + +- [hhatto/autopep8#227][227]: Create diff for a specific pep8 issue + User should have a choice of selecting the issue he wants to generate a fix + for. This issue is related to the above issue since the fix for this issue + will introduce a new setting `--column-range` in autopep8. Along with fixing these issues, new bears including `OutdatedDependencyBear`, `RegexLintBear`, `FileModeBear` and `RequirementsCheckBear` @@ -66,7 +66,7 @@ The bear will require changes to the existing [package_manager][package_manager] with the help of new additions to the requirement classes. - For pip requirements, we could use the PyPI JSON API which provides the - version info by specifying the `package_name`. + version info by specifying the `package_name`. ```py from xmlrpc import client @@ -84,7 +84,7 @@ class PipRequirement(PackageRequirement): ``` - For npm dependencies, we can use the npm cli command `npm outdated` which - produces an output as below: + produces an output as below: ``` $ npm outdated diff --git a/cEP-0036.md b/cEP-0036.md index 3370bf739..ed732b09a 100644 --- a/cEP-0036.md +++ b/cEP-0036.md @@ -32,16 +32,16 @@ changes to upgrade it to v12. ## Changes to IGitt -* [Docstrings, wherever missing, need to be added to the IGitt -codebase][2]. -* IGitt documentation needs to be updated to reflect changes occurring due to -migration from Gitmate to coala organisation. Also, a section can be added on -how IGitt codebase is structured, examples can be written for those actions -missing currently like notifications, reactions etc, and a section on how to -use and set up and use for developers can be added too. -* Cron jobs are currently missing for different versions of python, so they -need to be added after adding support for latest versions. -* coala-IGitt needs to be published as a separate package. +- [Docstrings, wherever missing, need to be added to the IGitt + codebase][2]. +- IGitt documentation needs to be updated to reflect changes occurring due to + migration from Gitmate to coala organisation. Also, a section can be added on + how IGitt codebase is structured, examples can be written for those actions + missing currently like notifications, reactions etc, and a section on how to + use and set up and use for developers can be added too. +- Cron jobs are currently missing for different versions of python, so they + need to be added after adding support for latest versions. +- coala-IGitt needs to be published as a separate package. ## Changes to Gitmate @@ -56,33 +56,33 @@ Gitmate. Multiple improvements can be made to the frontend. These include **UI changes** such as: -* Remove the use of German -* Remove the list of sponsor companies -* Revamp landing page -* Fix 'copy plugin settings': dropdown should not show 'suggested password' -* Make spinner go away after activating gitmate on a repository -* Fix buggy slide toggles -* [Fix alignment of toggles for suboptions][3] -* Allow user to go to ‘/repo/n’ by click on repository card -* [Add repository name in title][4] -* [Use material chips in text areas][5] -* [Make ‘copy plugin’ bar longer][6] -* Improve design of repository cards +- Remove the use of German +- Remove the list of sponsor companies +- Revamp landing page +- Fix 'copy plugin settings': dropdown should not show 'suggested password' +- Make spinner go away after activating gitmate on a repository +- Fix buggy slide toggles +- [Fix alignment of toggles for suboptions][3] +- Allow user to go to ‘/repo/n’ by click on repository card +- [Add repository name in title][4] +- [Use material chips in text areas][5] +- [Make ‘copy plugin’ bar longer][6] +- Improve design of repository cards **Frontend documentation** can be improved and the following changes can be made: -* Improve developers section - Fix broken links, improve part about setting -environment variables -* Complete FAQ.md -* Add a section on how the Gitmate codebase is structured. +- Improve developers section - Fix broken links, improve part about setting + environment variables +- Complete FAQ.md +- Add a section on how the Gitmate codebase is structured. ### Changes to backend -* Docstrings, wherever missing, need to be added to the backend codebase. -* The **‘ack’**, **‘approver’**, and **‘auto label pending or wip’ plugins** -need to be merged into [a][7] [combined][8] ['review'][9] [plugin][10] -. The models for the combined review plugins should be similar to this: +- Docstrings, wherever missing, need to be added to the backend codebase. +- The **‘ack’**, **‘approver’**, and **‘auto label pending or wip’ plugins** + need to be merged into [a][7] [combined][8] ['review'][9] [plugin][10]. + The models for the combined review plugins should be similar to this: ```python class Settings(SettingsBase): @@ -137,9 +137,9 @@ class MergeRequestModel(models.Model): def igitt_pr(self): return self.repo.igitt_repo.get_mr(self.number) ``` -* The rebaser plugin needs to be changed to ensure that the [proper labels][11] are -applied when rebase, fastforward, or merge is performed. The -following functions must be added in `responders.py` of rebaser plugin: +- The rebaser plugin needs to be changed to ensure that the [proper labels][11] are + applied when rebase, fastforward, or merge is performed. The + following functions must be added in `responders.py` of rebaser plugin: ```python def verify_appropriate_label(cmd: str, @@ -169,13 +169,23 @@ def handle_command_fail(pr: MergeRequest, ``` [1]: https://summerofcode.withgoogle.com/projects/#6263057774280704 + [2]: https://gitlab.com/gitmate/open-source/IGitt/-/issues/146 + [3]: https://gitlab.com/gitmate/open-source/gitmate-2-frontend/-/issues/104 + [4]: https://gitlab.com/gitmate/open-source/gitmate-2-frontend/-/issues/83 + [5]: https://gitlab.com/gitmate/open-source/gitmate-2-frontend/-/issues/81 + [6]: https://gitlab.com/gitmate/open-source/gitmate-2-frontend/-/issues/66 + [7]: https://gitlab.com/gitmate/open-source/gitmate-2/-/issues/242 + [8]: https://gitlab.com/gitmate/open-source/gitmate-2/-/issues/375 + [9]: https://gitlab.com/gitmate/open-source/gitmate-2/-/issues/376 + [10]: https://gitlab.com/gitmate/open-source/gitmate-2/-/issues/243 + [11]: https://gitlab.com/gitmate/open-source/gitmate-2/-/issues/144