Release Management
This document provides comprehensive guidance for deploying and managing releases of the PermitProof application across different environments.
Overview
The PermitProof application consists of five main deployment components:
- gRPC Backend Service (Cloud Run) - Core business logic and API endpoints
- Cloud Run Jobs - Long-running task processors (plan ingestion, code applicability, compliance reports)
- WebSocket Chat Service (Cloud Run) - Real-time chat streaming with ADK agent (Issue #270)
- ESPv2 API Gateway (Cloud Endpoints) - Proxy and authentication layer
- Angular Frontend (Firebase Hosting) - Web application UI
Full-Stack Deployment
The cli/sdlc/full-stack-deploy.sh script orchestrates deployment of all components in the correct order.
Quick Start
# Deploy entire stack to dev environment (from project root)
./cli/sdlc/full-stack-deploy.sh dev
# Deploy to test environment without rebuilding
./cli/sdlc/full-stack-deploy.sh test --skip-build
# Force deploy to prod from dirty working directory
./cli/sdlc/full-stack-deploy.sh prod --force
Available Options
./cli/sdlc/full-stack-deploy.sh [env] [options]
# Options:
# --skip-build Skip build steps and deploy existing artifacts
# --skip-tests Skip unit tests during build (default: enabled)
# --force Allow deployment from dirty Git working directory
# --skip-grpc Skip gRPC backend service deployment
# --skip-jobs Skip Cloud Run Jobs deployment
# --skip-websocket Skip WebSocket chat service deployment
# --skip-endpoints Skip ESPv2 API Gateway deployment
# --skip-frontend Skip Angular frontend deployment
# --help, -h Show help message
Selective Deployment
Deploy only specific components:
# Deploy only backend components (skip frontend)
./cli/sdlc/full-stack-deploy.sh dev --skip-frontend
# Deploy only frontend and API gateway (skip backend builds)
./cli/sdlc/full-stack-deploy.sh demo --skip-grpc --skip-jobs
# Update only the frontend
./cli/sdlc/full-stack-deploy.sh test --skip-grpc --skip-jobs --skip-endpoints
Prerequisites
Before running full-stack deployment:
- Java 17+ installed
- Apache Maven installed
- Node.js and npm installed
- Google Cloud SDK (gcloud) installed and authenticated
- Firebase CLI installed and authenticated
- Environment configuration files in
env/{env}/ - Clean Git working directory (unless
--forceis used)
Deployment Order
Components are deployed in this order to respect dependencies:
- gRPC Backend Service: Core business logic and API endpoints
- Cloud Run Jobs: Long-running task processors that depend on the same codebase
- ESPv2 API Gateway: Proxy and authentication layer that routes to the gRPC service
- Angular Frontend: Web application that consumes the API through the gateway
Timing and Progress
The script provides detailed timing information:
🎉 Full-Stack Deployment Summary
================================
Environment: dev
Total Time: 420s
⏱️ Component Timing Breakdown:
🚀 gRPC Backend Service: 180s
🏗️ Cloud Run Jobs: 120s
🌐 ESPv2 API Gateway: 60s
🖥️ Angular Frontend: 60s
Error Handling
The script uses strict error handling (set -euo pipefail) and will:
- Stop deployment if any component fails
- Validate all prerequisites before starting
- Check Git working directory status (unless
--forceis used) - Provide clear error messages for troubleshooting
gRPC Backend Service Deployment
The gRPC backend service deployment is handled by cli/sdlc/cloud-run-grpc/deploy.sh.
Quick Deployment
# Deploy to dev environment
./cli/sdlc/cloud-run-grpc/deploy.sh dev
# Deploy to test environment
./cli/sdlc/cloud-run-grpc/deploy.sh test
# Deploy to prod environment
./cli/sdlc/cloud-run-grpc/deploy.sh prod
Deployment Options
# Available options
./cli/sdlc/cloud-run-grpc/deploy.sh [env] [options]
# Options:
# --skip-build Deploy most recently built local image without rebuilding
# --skip-tests Skip unit tests during build (default: enabled)
# --force Allow deployment from dirty Git working directory
# --help, -h Show help message
# Examples:
./cli/sdlc/cloud-run-grpc/deploy.sh dev --skip-build
./cli/sdlc/cloud-run-grpc/deploy.sh prod --force
Deployment Process
The script performs the following steps:
- Validates Git working directory (unless
--forceis used) - Builds the application using Maven (unless
--skip-buildis used)- Uses thin JAR + separate dependencies approach
- Skips tests by default for speed
- Builds Docker image tagged with Git commit SHA
- Pushes image to shared Artifact Registry (
construction-code-expert-repo) - Deploys to Cloud Run with the new image
Resource Configuration
Current gRPC service configuration:
- Memory: 32Gi (increased for large PDF processing)
- CPU: 8 cores
- Timeout: 3600s (1 hour)
- Concurrency: 8 requests per instance
- Min Instances: 0 (scales to zero)
See GitHub Issue #191 for deployment optimization details.
Test deployment via gRPC calls
Test the deployed gRPC service directly:
# Set the gRPC server host and generate bearer token
GRPC_SERVER_HOST=construction-code-expert-dev-856365345080.us-central1.run.app
CODEPROOF_FIREBASE_TOKEN=$(./firebase-token-generator/generate-token.sh sanchos101@gmail.com)
# Test GetArchitecturalPlan RPC (with authentication)
grpcurl -import-path src/main/proto \
-proto src/main/proto/api.proto \
-H "authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14"}' \
${GRPC_SERVER_HOST}:443 \
org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlan
# Test PDF retrieval (note: use -max-msg-sz for large PDFs)
grpcurl -import-path src/main/proto \
-proto src/main/proto/api.proto \
-H "authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14", "page_number": 1}' \
-max-msg-sz $((10 * 1024 * 1024)) \
${GRPC_SERVER_HOST}:443 \
org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlanPagePdf
Note: When calling gRPC directly, only the authorization header is needed. When calling via ESPv2 proxy or REST API, both Authorization and x-original-authorization headers are required.
Cloud Run Jobs Deployment
Cloud Run Jobs handle long-running tasks that exceed the 15-minute Cloud Run Service timeout. The deployment is managed by cli/sdlc/cloud-run-job/deploy.sh.
Job Types
The following job types are available:
| Job Type | Cloud Run Job Name | Main Class | Purpose |
|---|---|---|---|
plan-ingestion | plan-ingestion-processor-{env} | PlanIngestionProcessorJobMain | PDF page extraction and processing |
code-applicability | code-applicability-processor-{env} | CodeApplicabilityProcessorJobMain | Code section applicability analysis |
compliance-report | compliance-report-processor-{env} | ComplianceReportProcessorJobMain | Compliance report generation |
Deploy All Jobs
# Deploy ALL job types to dev environment
./cli/sdlc/cloud-run-job/deploy.sh dev
# Deploy all jobs to test environment
./cli/sdlc/cloud-run-job/deploy.sh test
Deploy Specific Jobs
# Deploy only plan ingestion job to dev
./cli/sdlc/cloud-run-job/deploy.sh plan-ingestion dev
# Deploy only code applicability job to test
./cli/sdlc/cloud-run-job/deploy.sh code-applicability test
# Deploy only compliance report job to prod
./cli/sdlc/cloud-run-job/deploy.sh compliance-report prod
Deployment Options
# Available options
./cli/sdlc/cloud-run-job/deploy.sh [task-type] [env] [options]
# Options:
# --skip-build Deploy most recently built local images without rebuilding
# --skip-tests Skip unit tests during build (default: enabled)
# --force Allow deployment from dirty Git working directory
# --help, -h Show help message
# Examples:
./cli/sdlc/cloud-run-job/deploy.sh plan-ingestion dev --skip-build
./cli/sdlc/cloud-run-job/deploy.sh code-applicability prod --force
./cli/sdlc/cloud-run-job/deploy.sh dev --skip-tests
Job Resource Configuration
| Job Type | Memory | CPU | Timeout | Typical Duration |
|---|---|---|---|---|
| plan-ingestion | 4Gi | 8 cores | 3600s (1 hour) | 15-20 minutes |
| code-applicability | 4Gi | 2 cores | 1800s (30 min) | 2-15 minutes |
| compliance-report | 4Gi | 2 cores | 1800s (30 min) | 5-20 minutes |
Build Process
The script uses Maven profiles to build job-specific JARs:
# Build with cloud-run-jobs profile
mvn clean package -DskipTests -P cloud-run-jobs
This creates three separate job JARs:
construction-code-expert-{version}-plan-ingestion-job.jarconstruction-code-expert-{version}-code-applicability-job.jarconstruction-code-expert-{version}-compliance-report-job.jar
Verify Deployment
# List all Cloud Run Jobs
gcloud run jobs list --region=us-central1 --project=construction-code-expert-dev
# Describe specific job
gcloud run jobs describe plan-ingestion-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev
Test Triggering Cloud Run Jobs
Test individual jobs manually:
# Test plan ingestion job
gcloud run jobs execute plan-ingestion-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--args="test-task-ingestion,R2024.0091-2024-10-14,21_Wiggins_DCA_Progress_Set_10142024-1.pdf,6,true,true"
# Test code applicability job
gcloud run jobs execute code-applicability-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--args="test-task-applicability,R2024.0091-2024-10-14,6,2217,5"
# Test compliance report job
gcloud run jobs execute compliance-report-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--args="test-task-compliance,R2024.0091-2024-10-14,7,2217,IBC2021P2_Ch11_Sec1101"
ESPv2 API Gateway Deployment
The ESPv2 (Envoy Proxy) deployment is handled by env/deploy-endpoints.sh.
Quick Deployment
cd env
./deploy-endpoints.sh dev
Prerequisites
# Enable required APIs
gcloud services enable servicecontrol.googleapis.com --project=construction-code-expert-dev
gcloud services enable endpoints.googleapis.com --project=construction-code-expert-dev
# Grant Service Control IAM role to service account
gcloud projects add-iam-policy-binding construction-code-expert-dev \
--member="serviceAccount:cce-app-service@construction-code-expert-dev.iam.gserviceaccount.com" \
--role="roles/servicemanagement.serviceController"
Test ESPv2 deployment via RPC calls to Geteway
Quick smoke test to verify ESPv2 is routing to the backend:
# Set the ESPv2 service host and generate bearer token
ESP_SERVICE_HOST=construction-code-expert-esp2-dev-6yieikr6ca-uc.a.run.app
CODEPROOF_FIREBASE_TOKEN=$(./firebase-token-generator/generate-token.sh sanchos101@gmail.com)
# Test via gRPC through ESPv2
grpcurl -import-path src/main/proto \
-proto src/main/proto/api.proto \
-H "authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-H "x-original-authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14"}' \
${ESP_SERVICE_HOST}:443 \
org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlan
# Test via REST API through ESPv2
curl -XPOST -H "Content-Type: application/json" \
-H "Authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-H "x-original-authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14"}' \
https://${ESP_SERVICE_HOST}/org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlan
Note: ESPv2 proxy calls require both Authorization and x-original-authorization headers.
📖 For comprehensive REST API testing examples, see REST API Testing
Build ESPv2 Image (GitOps Workflow)
For GitOps workflows where you want to build the ESPv2 image without immediately deploying it, use env/build-espv2-image.sh. This separates the build phase from deployment, allowing the artifact to be built once and promoted across environments.
When to Use
| Script | Use Case |
|---|---|
deploy-endpoints.sh | Quick iteration: build and deploy in one step |
build-espv2-image.sh | GitOps: build image, store in central registry, deploy later |
Usage
cd env
./build-espv2-image.sh <environment>
# Example: ./build-espv2-image.sh stg
⚠️ Environment-Specific Configuration: The ESPv2 deployment requires environment-specific settings:
Parameter Description Example (dev) ESPv2 Hostname The Cloud Run hostname of the proxy itself construction-code-expert-esp2-dev-xxx.a.run.appBackend Address The gRPC backend this proxy routes to grpcs://construction-code-expert-dev-xxx.a.run.appFirebase Issuer Auth token issuer for JWT validation https://securetoken.google.com/construction-code-expert-devFirebase Audience Expected audience in JWT tokens construction-code-expert-devEndpoint Config ID Unique Cloud Endpoints configuration version 2024-12-15r0Current approach: Each environment gets a custom-built image with config baked in.
TODO (#327): Use the stock ESPv2 base image (
gcr.io/endpoints-release/endpoints-runtime-serverless) with--rollout_strategy=managedpassed viaESPv2_ARGS. This would allow:
- Same base image across environments
- Config passed via environment variables at deploy time
- Auto-fetch of latest Cloud Endpoints config at startup
The Cloud Endpoints service registration (
api_config.yaml) will still be per-environment since it contains the backend address and Firebase project.
What It Does
- Generates Protobuf descriptors from
src/main/proto/*.proto - Deploys Endpoint Config to Cloud Endpoints (Service Management API)
- Builds ESPv2 Docker image using Cloud Build
- Promotes image to Central Artifact Registry (
construction-code-expert-repo)
Output
✅ Build & Promotion Complete!
Local Env Image: gcr.io/construction-code-expert-stg/endpoints-runtime-serverless:...
Central Repo Image: us-central1-docker.pkg.dev/construction-code-expert-repo/custom-docker-image-repo/construction-code-expert-espv2:<config-id>
You can now update your variables to use: '<central-image-uri>'
Terraform Integration
The promoted image can be referenced in Terraform configurations or used with gcloud run deploy:
# Deploy the pre-built image to Cloud Run
gcloud run deploy construction-code-expert-esp2-stg \
--image=us-central1-docker.pkg.dev/construction-code-expert-repo/custom-docker-image-repo/construction-code-expert-espv2:<config-id> \
--region=us-central1 \
--project=construction-code-expert-stg
Frontend Deployment
The Angular frontend deployment is handled by web-ng-m3/deploy.sh.
Quick Deployment
cd web-ng-m3
./deploy.sh dev
⚠️ IMPORTANT: This script MUST be run from within the Devcontainer (See Issue #399). The Angular build process uses
esbuild, which relies on platform-specific native binaries. Running this on a host machine (e.g., macOS) afternpm installwas run in the container (Linux) will cause a platform mismatch error.
Build and Deploy Process
The script performs:
- Generates gRPC-Web sources from proto files
- Builds Angular application for the target environment
- Deploys to Firebase Hosting with environment-specific configuration
- Deploys Firestore indexes and rules (via dedicated script)
Manual Deployment Steps
cd web-ng-m3
# Load environment variables
source ../env/dev/setvars.sh
source ../env/dev/firebase/m3/setvars.sh
# Install dependencies (if needed)
npm install
# Generate gRPC-Web sources
../cli/sdlc/utils/generate-grpc-web-sources.sh
# Build for specific environment
npm run build:dev
# Deploy to Firebase hosting
firebase deploy --project=construction-code-expert-dev --only hosting:dev
# Deploy Firestore indexes and rules
../cli/sdlc/firestore/deploy.sh dev
Firestore Deployment
The Firestore deployment is handled by cli/sdlc/firestore/deploy.sh and manages database indexes and security rules.
Quick Deployment
# Deploy both indexes and rules to dev environment
./cli/sdlc/firestore/deploy.sh dev
# Deploy only Firestore indexes (skip rules)
./cli/sdlc/firestore/deploy.sh test --indexes-only
# Deploy only Firestore rules (skip indexes)
./cli/sdlc/firestore/deploy.sh prod --rules-only
What Gets Deployed
- Firestore Indexes: Composite indexes from
web-ng-m3/firestore.indexes.jsonfor efficient querying - Firestore Rules: Security rules from
web-ng-m3/firestore.rulesfor access control
When to Use Firestore Deployment
- After adding new composite indexes to
firestore.indexes.json - After updating security rules in
firestore.rules - When setting up a new environment
- When experiencing Firestore query performance issues
- When seeing "query requires an index" errors in the console
Integration with Other Deployments
- Full-stack deployment: Automatically includes Firestore deployment
- Frontend deployment: Automatically includes Firestore deployment
- Independent deployment: Can be run separately for targeted updates
Manual Firestore Deployment Steps
# Load environment variables
source env/dev/setvars.sh
# Deploy both indexes and rules
./cli/sdlc/firestore/deploy.sh dev
# Or deploy individually
firebase deploy --project=construction-code-expert-dev --only firestore:indexes
firebase deploy --project=construction-code-expert-dev --only firestore:rules
Environment-Specific Configurations
Each environment has its own configuration in env/{env}/:
env/
├── dev/
│ ├── setvars.sh
│ ├── gcp/cloud-run/grpc/
│ │ ├── setvars.sh
│ │ └── vars.yaml
│ ├── gcp/cloud-run/endpoints/
│ │ ├── setvars.sh
│ │ └── api_config.yaml
│ └── firebase/m3/
│ ├── setvars.sh
│ └── firebaseConfig.json
├── test/
├── demo/
└── prod/
Loading Environment Variables
# Load environment-specific variables
source env/dev/setvars.sh
# For gRPC deployments
source env/dev/gcp/cloud-run/grpc/setvars.sh
# For ESPv2 deployments
source env/dev/gcp/cloud-run/endpoints/setvars.sh
# For frontend deployments
source env/dev/firebase/m3/setvars.sh
Deployment Best Practices
Pre-Deployment Checklist
- All tests passing locally
- Proto files regenerated if modified
- Environment variables configured
- Git working directory clean (or use
--force) - Service account permissions verified
- Target environment resources provisioned
Deployment Strategy
Development Environment (dev):
- Deploy frequently with
--skip-testsfor speed - Use
--skip-buildfor quick config-only changes - Test new features before promoting to other environments
Test Environment (test):
- Used for AI SWE Agent integration testing
- Deploy after dev testing is complete
- Run full test suite before promoting
Demo Environment (demo):
- Deploy stable releases for customer demos
- Ensure all features are fully tested
- Use production-like data and configuration
Production Environment (prod):
- Deploy only after thorough testing in test/demo
- Never use
--forceflag - Always deploy during maintenance windows
- Have rollback plan ready
Rollback Procedure
If a deployment fails or causes issues:
# 1. Identify the previous working image tag
gcloud run services describe construction-code-expert-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--format='value(status.traffic[0].revisionName)'
# 2. Deploy previous image
./cli/sdlc/cloud-run-grpc/deploy.sh dev --skip-build
# 3. For frontend rollback, redeploy previous version
cd web-ng-m3
firebase hosting:rollback --project=construction-code-expert-dev
Monitoring and Verification
Health Checks
TODO: Health endpoint implementation needs investigation. See GitHub Issue #222
# Check gRPC service health
curl https://construction-code-expert-dev-856365345080.us-central1.run.app/health
# Check ESPv2 gateway health
curl https://construction-code-expert-esp2-dev-6yieikr6ca-uc.a.run.app/health
Log Monitoring
# Get recent Cloud Run logs
./get-cloud-run-logs.sh --env=dev --minutes=30
# Query specific job logs
gcloud logging read "resource.type=cloud_run_job AND resource.labels.job_name=plan-ingestion-processor-dev" \
--project=construction-code-expert-dev --limit=50
Service Status
# Check Cloud Run service status
gcloud run services describe construction-code-expert-dev \
--region=us-central1 \
--project=construction-code-expert-dev
# Check Cloud Run job status
gcloud run jobs list --region=us-central1 --project=construction-code-expert-dev
# Check Firebase hosting status
firebase hosting:channel:list --project=construction-code-expert-dev
Troubleshooting
Common Issues
1. Docker Build Failures
- Ensure Maven build completed successfully
- Check that thin JAR and dependencies exist in
target/ - Verify Dockerfile paths are correct
2. Image Push Failures
- Authenticate Docker:
gcloud auth configure-docker us-central1-docker.pkg.dev - Verify Artifact Registry permissions
- Check network connectivity
3. Deployment Timeouts
- Increase timeout in deployment script
- Check Cloud Run service logs for errors
- Verify service account has required permissions
4. Frontend Build Failures
- Regenerate gRPC-Web sources
- Clear node_modules and reinstall:
rm -rf node_modules && npm install - Check environment variables are set correctly
Getting Help
# View help for any deployment script
./cli/sdlc/full-stack-deploy.sh --help
./cli/sdlc/cloud-run-grpc/deploy.sh --help
./cli/sdlc/cloud-run-job/deploy.sh --help
Related Documentation
- Environment Provisioning - Setting up new environments from scratch
- SDLC Scripts - Overview of all deployment and utility scripts
- Background Tasks Architecture - Understanding Cloud Run Jobs
- Developer Playbook - General development workflows