From 9a0118c603e8a773ca468c2382204e475e5bf7d4 Mon Sep 17 00:00:00 2001 From: Zach White Date: Tue, 10 Aug 2021 07:47:53 -0700 Subject: [PATCH 1/2] Architecture documentation for Configurator and API (#13935) * Architecture documentation for the configurator and api * Apply suggestions from code review Co-authored-by: James Young <18669334+noroadsleft@users.noreply.github.com> Co-authored-by: James Young <18669334+noroadsleft@users.noreply.github.com> --- docs/_summary.md | 1 + docs/configurator_architecture.md | 61 +++++++++++++++++++++++++++++++ docs/configurator_diagram.drawio | 1 + docs/configurator_diagram.svg | 3 ++ 4 files changed, 66 insertions(+) create mode 100644 docs/configurator_architecture.md create mode 100644 docs/configurator_diagram.drawio create mode 100644 docs/configurator_diagram.svg diff --git a/docs/_summary.md b/docs/_summary.md index 9798ef512..8347805a2 100644 --- a/docs/_summary.md +++ b/docs/_summary.md @@ -19,6 +19,7 @@ * [Overview](newbs_building_firmware_configurator.md) * [Step by Step](configurator_step_by_step.md) * [Troubleshooting](configurator_troubleshooting.md) + * [Architecture](configurator_architecture.md) * QMK API * [Overview](api_overview.md) * [API Documentation](api_docs.md) diff --git a/docs/configurator_architecture.md b/docs/configurator_architecture.md new file mode 100644 index 000000000..0d7fc8a73 --- /dev/null +++ b/docs/configurator_architecture.md @@ -0,0 +1,61 @@ +# QMK Configurator Architecture + +This page describes the web architecture behind QMK Configurator at a high level. If you are interested in the architecture of the QMK Configurator code itself you should start at the [qmk_configurator](https://github.com/qmk/qmk_configurator) repository. + +# Overview + +![QMK Configurator Architecture Diagram](configurator_diagram.svg) + +# Detailed Description + +QMK Configurator is a [Single Page Application](https://en.wikipedia.org/wiki/Single-page_application) that allows users to create custom keymaps for their QMK-compatible keyboard. They can export JSON representation of their keymaps and compile firmware binaries that can be flashed to their keyboard using a tool like [QMK Toolbox](https://github.com/qmk/qmk_toolbox). + +Configurator gets metadata about keyboards from the Keyboard Metadata store and submits compile requests to the QMK API. The results of those compile requests will be made available on [Digital Ocean Spaces](https://www.digitalocean.com/products/spaces/), an S3-compatible data store. + +## Configurator Frontend + +Address: + +The [Configurator Frontend](https://config.qmk.fm) is compiled into a set of static files that are served by Github Pages. This action happens every time a commit is pushed to the [qmk_configurator `master`](https://github.com/qmk/qmk_configurator) branch. You can view the status of these jobs on the [qmk_configurator actions tab](https://github.com/qmk/qmk_configurator/actions/workflows/build.yml). + +## Keyboard Metadata + +Address: + +The Keyboard Metadata is generated every time a keyboard in [qmk_firmware](https://github.com/qmk/qmk_firmware) changes. The resulting JSON files are uploaded to Spaces and used by Configurator to generate UI for each keyboard. You can view the status of this job on the [qmk_firmware actions tab](https://github.com/qmk/qmk_firmware/actions/workflows/api.yml). If you are a QMK Collaborator you can manually run this job using the `workflow_dispatch` event trigger. + +## QMK API + +Address: + +The QMK API accepts `keymap.json` files for compilation. These are the same files you can use directly with `qmk compile` and `qmk flash`. When a `keymap.json` is submitted the browser will poll the status of the job periodically (every 2 seconds or longer, preferably) until the job has completed. The final status JSON will contain pointers to source and binary downloads for the keymap. + +QMK API always presents the source and binary downloads side-by-side to comply with the GPL. + +There are 3 non-error status responses from the API- + +1. Compile Job Queued +2. Compile Job Running +3. Compile Job Finished + +### Compile Job Queued + +This status indicates that the job has not yet been picked up by a [QMK Compiler](#qmk-compiler) node. Configurator shows this status as "Waiting for an oven". + +### Compile Job Running + +This status indicates that the job has started compiling. Configurator shows this status as "Baking". + +### Compile Job Finished + +This status indicates that the job has completed. There will be keys in the status JSON for source and binary downloads. + +## Redis/RQ + +QMK API uses RQ to distribute jobs to the available [QMK Compiler](#qmk-compiler) nodes. When a `keymap.json` is received it's put into the RQ queue, where a `qmk_compiler` node will pick it up from. + +## QMK Compiler + +[QMK Compiler](https://github.com/qmk/qmk_compiler) is what actually performs the compilation of the `keymap.json`. It does so by checking out the requested `qmk_firmware` branch, running `qmk compile keymap.json`, and then uploading the resulting source and binary to Digital Ocean Spaces. + +When users download their source/binary, API will redirect them to the authenticated Spaces download URL. diff --git a/docs/configurator_diagram.drawio b/docs/configurator_diagram.drawio new file mode 100644 index 000000000..091a3a76b --- /dev/null +++ b/docs/configurator_diagram.drawio @@ -0,0 +1 @@ +5VvbcqM4EP2a1O4+hOLqy2Ni5zKX1CTxzszOU0oG2dZEIBZEYu/XbwuEDQg7csZ2vFlXjQca0RLdR+eohXPiDML5VYLi2Q0LMD2xzWB+4gxPbNvqezb8JywLaTFtr7BMExJI28owIv/gsqG0ZiTAaa0hZ4xyEteNPosi7POaDSUJe643mzBa7zVGU6wYRj6iqvU7CfissPY8c2W/xmQ6K3u2THklRGVjaUhnKGDPFZNzceIMEsZ4cRTOB5iK6JVxKe67XHN1ObAER1znhgfT76Zzxxp3Z49j5+7zjw/z+NSS+XlCNJNPLEfLF2UIcAARkacs4TM2ZRGiFyvrecKyKMCiHxPOVm0+MxaD0QLjT8z5QqYXZZyBacZDKq/iKDgTyYLTiEW4sFwSSqXLAKWz3L9orD64jEXKssTHG57WlQBCyRTzTVGRDsWTV3qQcb3CLMQ8WUCDBFPEyVMdK0hCbrpst8oKHMjEbJEk9+UckTDHcTWm0jQk4RR6pWQM3z4l8QNKuDhkYZxxnKRw/J0ljymHJ2HRg2X35vDPiKPp2kA/4YTj+cbIyKunjmv0zMpHeljSQXH6XJla0jSrzKrStvPIei9Hdgm9HNoZpyTCgyXZmPWYI0qmERz7EDGcgIGiMaa3LCUittULIoQEaOZzo8GYcc7CSoMz6ZKLqXQOJBKLgYXzqSBc4xmPKcy11JgSPsvG0GLCIn6JQkJFiK8xfcLCjbwgJyDM+uJ8wChL8ud0ivSILnjCHnHblQlMyIp9kn/ADkMJCK55G14OL4YXmybrFhhyOzXQtKEGdMVwVeBUzDvHTjmKjdS54jWfojQlfpP3YCR/CRwZXnn6o3ptOJcgK84W8gwma8JVz7lZ0qZVI9Ff401PkzddTdqsJM5rme6lTZtdZQ+3jMCTLXFjlRkqyaZJI8Vzy7uqytl01Gk4chqOisAojnJYLR/7F5BWTpkV0gZUTDjgbnOUxTEorqCosxO7QyGG5+MEjqbiaDBLoPsTGzo2L0mCJ2yuNhri9FHQi21+iehCAfHzjHA8ilEOkWegnTqGm0QVkiDIVwU5950j/3Garw9Kdsjd7YYZlmurkhlUYrDNFoB19qUnltPCCUW40xhFZcBnnIvl65noTChxNCFT4+/w0ZiEZXOwV+9Qk9bmd0SiKYTeNm+F9AM2RKTX3tno4iN6QiM/IbFYHXz7eqHeOJI8AeM2YbTQDL4fiuFnCeIs0RvmNUt54eWqUK1iwOnaZ28Asr7cfC0894E/q2PYehDs7w2CGqtFdbndqklVRaoI1BpN2tEaXVdryql2JGKzTKiEgvdasXHchqNDi01XAz/vpiK0tOHW0YTbYUpC6/1ULj9TuH0XlWZ93riOuhjotsz/rmf09pWkjpKkuvI/4sWYoSRIG+K/ks9PsgV4ucEcBYgjXV2ekCR8hoiq7a+uoeGZX+RN3PQ1BsdiwXB2+wG+h629rFR7SKDaRPSLj4VIm7n2ptsuG2voPKRIu7Zt9N9apHtbiXRr7VirASVHVgtAc4cFoDZNekclypZnWK5p9Tp9y+v1up1y73nRQMbWBWHdz3LP90ASbav14HuW6L4m9spsHotE95Us3d18Kkm2mS5KSZziik76lGXBy+XNLpTTq8O5Y6rKadkts3Rv27JlZ2uFE8VkrWTeX4z+nGS0DLSeWILDTYp3jyJ/Bqsf2xRrFXP4Bb6+3fy2tegdUuc6pqenct29pVFjkfryDulhtzDLWXskCnbarCtd75WadaoUlpqqBelBi0qzWDRINwzZanTkOObmkTVv8Pq1G+CgGMNuRVSl53csonZXV0R7RyWitrobIasPMMoCRGF4WY807b+PnD+OS3idXh347fuHrdrbt8qWu4+5WpzU1RcE81QocLTwHcMPIiMoUsJEPtI8+obPWqRZRlNK7+kLoqskejut3UWCeg1msh3L6PYrn54qsf2WbO3thYOjrpTeTGK1Oea4ikSQ/EaSX7t1qwiZfeC9W6ft9dO71TSn/D3YS3hz7KPSNEct3+/vlDyVciS2/FLOEnwYSWpskXS7ai3YOSjB2WqwcEDEm+9cjEwRuqaGfGTineJdhrOW3c/2AjApnO6/BNxBkpqVXrfz1pWes91Lx73KkPZ+UTkP31qHyiw26zFt2fEacGiWiPtWHXWNXvwuIIwpETNmDbPhuZhhAcvG+RWrJj3Wm2x99Vt+QXLYrS9XXdAVu4YDCCehIpztbHefRVH+a07dX22sc/efoLy+/daU5+pswh+I8vRXQt3joDyZVbeW015ztaxLgHU3/eZbnT3Tn6uuULYGwoF/z+lo7wYdF16AHes00GuuO/V3Qxui2bP0ULP1bqjTYK5yS2Gvm5vu/6sQ1IVzKa67KwTXEJK3GVu6KLWd14H0ZRjB6eovj4rmqz/gci7+BQ== \ No newline at end of file diff --git a/docs/configurator_diagram.svg b/docs/configurator_diagram.svg new file mode 100644 index 000000000..bcf0bf76d --- /dev/null +++ b/docs/configurator_diagram.svg @@ -0,0 +1,3 @@ + + +
Clients Supported:
Chrome, Firefox
Desktop Only
Clients Supported:...
https://config.qmk.fm
Single Page Site
JavaScript/VUE
Source: qmk/qmk_configurator
Host: Github Pages
https://config.qmk.fm...
https://keyboards.qmk.fm
Keyboard Metadata
Source: qmk/qmk_firmware
GH Action: Update API Data
Host: DigitalOcean Spaces
https://keyboards.qmk.fm...
QMK API
QMK API
https://api.qmk.fm
RESTful API
Source: qmk/qmk_api
Host: Rancher on DO VM's
https://api.qmk.fm...
Digital Ocean
Spaces
(S3)
Digital Ocean...
https://qmk-api.nyc3.cdn.digitaloceanspaces.com
Space: qmk-api
Host: Digital Ocean
https://qmk-api.nyc3.cdn.digitaloceanspaces.com...
RQ
RQ
Redis / RQ
Job Queue
Source: qmk/qmk_redis
Host: Rancher on DO VM's
Redis / RQ...
qmk_complier
qmk_complier
QMK Compiler
Job Runners
Source: qmk/qmk_compiler
Host: Rancher on DO VM's
QMK Compiler...Viewer does not support full SVG 1.1 \ No newline at end of file From ed84a4e7e3e65d1ef090117cdb9c6d70ed554a28 Mon Sep 17 00:00:00 2001 From: Zach White Date: Tue, 10 Aug 2021 07:48:22 -0700 Subject: [PATCH 2/2] Improve the qmk lint readme check (#13941) * improve lint readme check * improve if --- lib/python/qmk/cli/lint.py | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/lib/python/qmk/cli/lint.py b/lib/python/qmk/cli/lint.py index a164dba63..02b31fbc4 100644 --- a/lib/python/qmk/cli/lint.py +++ b/lib/python/qmk/cli/lint.py @@ -4,7 +4,7 @@ from milc import cli from qmk.decorators import automagic_keyboard, automagic_keymap from qmk.info import info_json -from qmk.keyboard import keyboard_completer +from qmk.keyboard import find_readme, keyboard_completer from qmk.keymap import locate_keymap from qmk.path import is_keyboard, keyboard @@ -31,7 +31,8 @@ def lint(cli): ok = True keyboard_path = keyboard(cli.config.lint.keyboard) keyboard_info = info_json(cli.config.lint.keyboard) - readme_path = keyboard_path / 'readme.md' + readme_path = find_readme(cli.config.lint.keyboard) + missing_readme_path = keyboard_path / 'readme.md' # Check for errors in the info.json if keyboard_info['parse_errors']: @@ -43,9 +44,9 @@ def lint(cli): cli.log.error('Warnings found when generating info.json (Strict mode enabled.)') # Check for a readme.md and warn if it doesn't exist - if not readme_path.exists(): + if not readme_path: ok = False - cli.log.error('Missing %s', readme_path) + cli.log.error('Missing %s', missing_readme_path) # Keymap specific checks if cli.config.lint.keymap: