Skip to main content

Release Management

This document provides comprehensive guidance for deploying and managing releases of the PermitProof application across different environments.

Overview

The PermitProof application consists of five main deployment components:

  1. gRPC Backend Service (Cloud Run) - Core business logic and API endpoints
  2. Cloud Run Jobs - Long-running task processors (plan ingestion, code applicability, compliance reports)
  3. WebSocket Chat Service (Cloud Run) - Real-time chat streaming with ADK agent (Issue #270)
  4. ESPv2 API Gateway (Cloud Endpoints) - Proxy and authentication layer
  5. Angular Frontend (Firebase Hosting) - Web application UI

Full-Stack Deployment

The cli/sdlc/full-stack-deploy.sh script orchestrates deployment of all components in the correct order.

Quick Start

# Deploy entire stack to dev environment (from project root)
./cli/sdlc/full-stack-deploy.sh dev

# Deploy to test environment without rebuilding
./cli/sdlc/full-stack-deploy.sh test --skip-build

# Force deploy to prod from dirty working directory
./cli/sdlc/full-stack-deploy.sh prod --force

Available Options

./cli/sdlc/full-stack-deploy.sh [env] [options]

# Options:
# --skip-build Skip build steps and deploy existing artifacts
# --skip-tests Skip unit tests during build (default: enabled)
# --force Allow deployment from dirty Git working directory
# --skip-grpc Skip gRPC backend service deployment
# --skip-jobs Skip Cloud Run Jobs deployment
# --skip-websocket Skip WebSocket chat service deployment
# --skip-endpoints Skip ESPv2 API Gateway deployment
# --skip-frontend Skip Angular frontend deployment
# --help, -h Show help message

Selective Deployment

Deploy only specific components:

# Deploy only backend components (skip frontend)
./cli/sdlc/full-stack-deploy.sh dev --skip-frontend

# Deploy only frontend and API gateway (skip backend builds)
./cli/sdlc/full-stack-deploy.sh demo --skip-grpc --skip-jobs

# Update only the frontend
./cli/sdlc/full-stack-deploy.sh test --skip-grpc --skip-jobs --skip-endpoints

Prerequisites

Before running full-stack deployment:

  • Java 17+ installed
  • Apache Maven installed
  • Node.js and npm installed
  • Google Cloud SDK (gcloud) installed and authenticated
  • Firebase CLI installed and authenticated
  • Environment configuration files in env/{env}/
  • Clean Git working directory (unless --force is used)

Deployment Order

Components are deployed in this order to respect dependencies:

  1. gRPC Backend Service: Core business logic and API endpoints
  2. Cloud Run Jobs: Long-running task processors that depend on the same codebase
  3. ESPv2 API Gateway: Proxy and authentication layer that routes to the gRPC service
  4. Angular Frontend: Web application that consumes the API through the gateway

Timing and Progress

The script provides detailed timing information:

🎉 Full-Stack Deployment Summary
================================
Environment: dev
Total Time: 420s

⏱️ Component Timing Breakdown:
🚀 gRPC Backend Service: 180s
🏗️ Cloud Run Jobs: 120s
🌐 ESPv2 API Gateway: 60s
🖥️ Angular Frontend: 60s

Error Handling

The script uses strict error handling (set -euo pipefail) and will:

  • Stop deployment if any component fails
  • Validate all prerequisites before starting
  • Check Git working directory status (unless --force is used)
  • Provide clear error messages for troubleshooting

gRPC Backend Service Deployment

The gRPC backend service deployment is handled by cli/sdlc/cloud-run-grpc/deploy.sh.

Quick Deployment

# Deploy to dev environment
./cli/sdlc/cloud-run-grpc/deploy.sh dev

# Deploy to test environment
./cli/sdlc/cloud-run-grpc/deploy.sh test

# Deploy to prod environment
./cli/sdlc/cloud-run-grpc/deploy.sh prod

Deployment Options

# Available options
./cli/sdlc/cloud-run-grpc/deploy.sh [env] [options]

# Options:
# --skip-build Deploy most recently built local image without rebuilding
# --skip-tests Skip unit tests during build (default: enabled)
# --force Allow deployment from dirty Git working directory
# --help, -h Show help message

# Examples:
./cli/sdlc/cloud-run-grpc/deploy.sh dev --skip-build
./cli/sdlc/cloud-run-grpc/deploy.sh prod --force

Deployment Process

The script performs the following steps:

  1. Validates Git working directory (unless --force is used)
  2. Builds the application using Maven (unless --skip-build is used)
    • Uses thin JAR + separate dependencies approach
    • Skips tests by default for speed
  3. Builds Docker image tagged with Git commit SHA
  4. Pushes image to shared Artifact Registry (construction-code-expert-repo)
  5. Deploys to Cloud Run with the new image

Resource Configuration

Current gRPC service configuration:

  • Memory: 32Gi (increased for large PDF processing)
  • CPU: 8 cores
  • Timeout: 3600s (1 hour)
  • Concurrency: 8 requests per instance
  • Min Instances: 0 (scales to zero)

See GitHub Issue #191 for deployment optimization details.

Test deployment via gRPC calls

Test the deployed gRPC service directly:

# Set the gRPC server host and generate bearer token
GRPC_SERVER_HOST=construction-code-expert-dev-856365345080.us-central1.run.app
CODEPROOF_FIREBASE_TOKEN=$(./firebase-token-generator/generate-token.sh sanchos101@gmail.com)

# Test GetArchitecturalPlan RPC (with authentication)
grpcurl -import-path src/main/proto \
-proto src/main/proto/api.proto \
-H "authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14"}' \
${GRPC_SERVER_HOST}:443 \
org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlan

# Test PDF retrieval (note: use -max-msg-sz for large PDFs)
grpcurl -import-path src/main/proto \
-proto src/main/proto/api.proto \
-H "authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14", "page_number": 1}' \
-max-msg-sz $((10 * 1024 * 1024)) \
${GRPC_SERVER_HOST}:443 \
org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlanPagePdf

Note: When calling gRPC directly, only the authorization header is needed. When calling via ESPv2 proxy or REST API, both Authorization and x-original-authorization headers are required.

Cloud Run Jobs Deployment

Cloud Run Jobs handle long-running tasks that exceed the 15-minute Cloud Run Service timeout. The deployment is managed by cli/sdlc/cloud-run-job/deploy.sh.

Job Types

The following job types are available:

Job TypeCloud Run Job NameMain ClassPurpose
plan-ingestionplan-ingestion-processor-{env}PlanIngestionProcessorJobMainPDF page extraction and processing
code-applicabilitycode-applicability-processor-{env}CodeApplicabilityProcessorJobMainCode section applicability analysis
compliance-reportcompliance-report-processor-{env}ComplianceReportProcessorJobMainCompliance report generation

Deploy All Jobs

# Deploy ALL job types to dev environment
./cli/sdlc/cloud-run-job/deploy.sh dev

# Deploy all jobs to test environment
./cli/sdlc/cloud-run-job/deploy.sh test

Deploy Specific Jobs

# Deploy only plan ingestion job to dev
./cli/sdlc/cloud-run-job/deploy.sh plan-ingestion dev

# Deploy only code applicability job to test
./cli/sdlc/cloud-run-job/deploy.sh code-applicability test

# Deploy only compliance report job to prod
./cli/sdlc/cloud-run-job/deploy.sh compliance-report prod

Deployment Options

# Available options
./cli/sdlc/cloud-run-job/deploy.sh [task-type] [env] [options]

# Options:
# --skip-build Deploy most recently built local images without rebuilding
# --skip-tests Skip unit tests during build (default: enabled)
# --force Allow deployment from dirty Git working directory
# --help, -h Show help message

# Examples:
./cli/sdlc/cloud-run-job/deploy.sh plan-ingestion dev --skip-build
./cli/sdlc/cloud-run-job/deploy.sh code-applicability prod --force
./cli/sdlc/cloud-run-job/deploy.sh dev --skip-tests

Job Resource Configuration

Job TypeMemoryCPUTimeoutTypical Duration
plan-ingestion4Gi8 cores3600s (1 hour)15-20 minutes
code-applicability4Gi2 cores1800s (30 min)2-15 minutes
compliance-report4Gi2 cores1800s (30 min)5-20 minutes

Build Process

The script uses Maven profiles to build job-specific JARs:

# Build with cloud-run-jobs profile
mvn clean package -DskipTests -P cloud-run-jobs

This creates three separate job JARs:

  • construction-code-expert-{version}-plan-ingestion-job.jar
  • construction-code-expert-{version}-code-applicability-job.jar
  • construction-code-expert-{version}-compliance-report-job.jar

Verify Deployment

# List all Cloud Run Jobs
gcloud run jobs list --region=us-central1 --project=construction-code-expert-dev

# Describe specific job
gcloud run jobs describe plan-ingestion-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev

Test Triggering Cloud Run Jobs

Test individual jobs manually:

# Test plan ingestion job
gcloud run jobs execute plan-ingestion-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--args="test-task-ingestion,R2024.0091-2024-10-14,21_Wiggins_DCA_Progress_Set_10142024-1.pdf,6,true,true"

# Test code applicability job
gcloud run jobs execute code-applicability-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--args="test-task-applicability,R2024.0091-2024-10-14,6,2217,5"

# Test compliance report job
gcloud run jobs execute compliance-report-processor-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--args="test-task-compliance,R2024.0091-2024-10-14,7,2217,IBC2021P2_Ch11_Sec1101"

ESPv2 API Gateway Deployment

The ESPv2 (Envoy Proxy) deployment is handled by env/deploy-endpoints.sh.

Quick Deployment

cd env
./deploy-endpoints.sh dev

Prerequisites

# Enable required APIs
gcloud services enable servicecontrol.googleapis.com --project=construction-code-expert-dev
gcloud services enable endpoints.googleapis.com --project=construction-code-expert-dev

# Grant Service Control IAM role to service account
gcloud projects add-iam-policy-binding construction-code-expert-dev \
--member="serviceAccount:cce-app-service@construction-code-expert-dev.iam.gserviceaccount.com" \
--role="roles/servicemanagement.serviceController"

Test ESPv2 deployment via RPC calls to Geteway

Quick smoke test to verify ESPv2 is routing to the backend:

# Set the ESPv2 service host and generate bearer token
ESP_SERVICE_HOST=construction-code-expert-esp2-dev-6yieikr6ca-uc.a.run.app
CODEPROOF_FIREBASE_TOKEN=$(./firebase-token-generator/generate-token.sh sanchos101@gmail.com)

# Test via gRPC through ESPv2
grpcurl -import-path src/main/proto \
-proto src/main/proto/api.proto \
-H "authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-H "x-original-authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14"}' \
${ESP_SERVICE_HOST}:443 \
org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlan

# Test via REST API through ESPv2
curl -XPOST -H "Content-Type: application/json" \
-H "Authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-H "x-original-authorization: Bearer ${CODEPROOF_FIREBASE_TOKEN}" \
-d '{"architectural_plan_id": "R2024.0091-2024-10-14"}' \
https://${ESP_SERVICE_HOST}/org.codetricks.construction.code.assistant.service.ArchitecturalPlanService/GetArchitecturalPlan

Note: ESPv2 proxy calls require both Authorization and x-original-authorization headers.

📖 For comprehensive REST API testing examples, see REST API Testing

Build ESPv2 Image (GitOps Workflow)

For GitOps workflows where you want to build the ESPv2 image without immediately deploying it, use env/build-espv2-image.sh. This separates the build phase from deployment, allowing the artifact to be built once and promoted across environments.

When to Use

ScriptUse Case
deploy-endpoints.shQuick iteration: build and deploy in one step
build-espv2-image.shGitOps: build image, store in central registry, deploy later

Usage

cd env
./build-espv2-image.sh <environment>
# Example: ./build-espv2-image.sh stg

⚠️ Environment-Specific Configuration: The ESPv2 deployment requires environment-specific settings:

ParameterDescriptionExample (dev)
ESPv2 HostnameThe Cloud Run hostname of the proxy itselfconstruction-code-expert-esp2-dev-xxx.a.run.app
Backend AddressThe gRPC backend this proxy routes togrpcs://construction-code-expert-dev-xxx.a.run.app
Firebase IssuerAuth token issuer for JWT validationhttps://securetoken.google.com/construction-code-expert-dev
Firebase AudienceExpected audience in JWT tokensconstruction-code-expert-dev
Endpoint Config IDUnique Cloud Endpoints configuration version2024-12-15r0

Current approach: Each environment gets a custom-built image with config baked in.

TODO (#327): Use the stock ESPv2 base image (gcr.io/endpoints-release/endpoints-runtime-serverless) with --rollout_strategy=managed passed via ESPv2_ARGS. This would allow:

  • Same base image across environments
  • Config passed via environment variables at deploy time
  • Auto-fetch of latest Cloud Endpoints config at startup

The Cloud Endpoints service registration (api_config.yaml) will still be per-environment since it contains the backend address and Firebase project.

What It Does

  1. Generates Protobuf descriptors from src/main/proto/*.proto
  2. Deploys Endpoint Config to Cloud Endpoints (Service Management API)
  3. Builds ESPv2 Docker image using Cloud Build
  4. Promotes image to Central Artifact Registry (construction-code-expert-repo)

Output

✅ Build & Promotion Complete!
Local Env Image: gcr.io/construction-code-expert-stg/endpoints-runtime-serverless:...
Central Repo Image: us-central1-docker.pkg.dev/construction-code-expert-repo/custom-docker-image-repo/construction-code-expert-espv2:<config-id>

You can now update your variables to use: '<central-image-uri>'

Terraform Integration

The promoted image can be referenced in Terraform configurations or used with gcloud run deploy:

# Deploy the pre-built image to Cloud Run
gcloud run deploy construction-code-expert-esp2-stg \
--image=us-central1-docker.pkg.dev/construction-code-expert-repo/custom-docker-image-repo/construction-code-expert-espv2:<config-id> \
--region=us-central1 \
--project=construction-code-expert-stg

Frontend Deployment

The Angular frontend deployment is handled by web-ng-m3/deploy.sh.

Quick Deployment

cd web-ng-m3
./deploy.sh dev

⚠️ IMPORTANT: This script MUST be run from within the Devcontainer (See Issue #399). The Angular build process uses esbuild, which relies on platform-specific native binaries. Running this on a host machine (e.g., macOS) after npm install was run in the container (Linux) will cause a platform mismatch error.

Build and Deploy Process

The script performs:

  1. Generates gRPC-Web sources from proto files
  2. Builds Angular application for the target environment
  3. Deploys to Firebase Hosting with environment-specific configuration
  4. Deploys Firestore indexes and rules (via dedicated script)

Manual Deployment Steps

cd web-ng-m3

# Load environment variables
source ../env/dev/setvars.sh
source ../env/dev/firebase/m3/setvars.sh

# Install dependencies (if needed)
npm install

# Generate gRPC-Web sources
../cli/sdlc/utils/generate-grpc-web-sources.sh

# Build for specific environment
npm run build:dev

# Deploy to Firebase hosting
firebase deploy --project=construction-code-expert-dev --only hosting:dev

# Deploy Firestore indexes and rules
../cli/sdlc/firestore/deploy.sh dev

Firestore Deployment

The Firestore deployment is handled by cli/sdlc/firestore/deploy.sh and manages database indexes and security rules.

Quick Deployment

# Deploy both indexes and rules to dev environment
./cli/sdlc/firestore/deploy.sh dev

# Deploy only Firestore indexes (skip rules)
./cli/sdlc/firestore/deploy.sh test --indexes-only

# Deploy only Firestore rules (skip indexes)
./cli/sdlc/firestore/deploy.sh prod --rules-only

What Gets Deployed

  • Firestore Indexes: Composite indexes from web-ng-m3/firestore.indexes.json for efficient querying
  • Firestore Rules: Security rules from web-ng-m3/firestore.rules for access control

When to Use Firestore Deployment

  • After adding new composite indexes to firestore.indexes.json
  • After updating security rules in firestore.rules
  • When setting up a new environment
  • When experiencing Firestore query performance issues
  • When seeing "query requires an index" errors in the console

Integration with Other Deployments

  • Full-stack deployment: Automatically includes Firestore deployment
  • Frontend deployment: Automatically includes Firestore deployment
  • Independent deployment: Can be run separately for targeted updates

Manual Firestore Deployment Steps

# Load environment variables
source env/dev/setvars.sh

# Deploy both indexes and rules
./cli/sdlc/firestore/deploy.sh dev

# Or deploy individually
firebase deploy --project=construction-code-expert-dev --only firestore:indexes
firebase deploy --project=construction-code-expert-dev --only firestore:rules

Environment-Specific Configurations

Each environment has its own configuration in env/{env}/:

env/
├── dev/
│ ├── setvars.sh
│ ├── gcp/cloud-run/grpc/
│ │ ├── setvars.sh
│ │ └── vars.yaml
│ ├── gcp/cloud-run/endpoints/
│ │ ├── setvars.sh
│ │ └── api_config.yaml
│ └── firebase/m3/
│ ├── setvars.sh
│ └── firebaseConfig.json
├── test/
├── demo/
└── prod/

Loading Environment Variables

# Load environment-specific variables
source env/dev/setvars.sh

# For gRPC deployments
source env/dev/gcp/cloud-run/grpc/setvars.sh

# For ESPv2 deployments
source env/dev/gcp/cloud-run/endpoints/setvars.sh

# For frontend deployments
source env/dev/firebase/m3/setvars.sh

Deployment Best Practices

Pre-Deployment Checklist

  • All tests passing locally
  • Proto files regenerated if modified
  • Environment variables configured
  • Git working directory clean (or use --force)
  • Service account permissions verified
  • Target environment resources provisioned

Deployment Strategy

Development Environment (dev):

  • Deploy frequently with --skip-tests for speed
  • Use --skip-build for quick config-only changes
  • Test new features before promoting to other environments

Test Environment (test):

  • Used for AI SWE Agent integration testing
  • Deploy after dev testing is complete
  • Run full test suite before promoting

Demo Environment (demo):

  • Deploy stable releases for customer demos
  • Ensure all features are fully tested
  • Use production-like data and configuration

Production Environment (prod):

  • Deploy only after thorough testing in test/demo
  • Never use --force flag
  • Always deploy during maintenance windows
  • Have rollback plan ready

Rollback Procedure

If a deployment fails or causes issues:

# 1. Identify the previous working image tag
gcloud run services describe construction-code-expert-dev \
--region=us-central1 \
--project=construction-code-expert-dev \
--format='value(status.traffic[0].revisionName)'

# 2. Deploy previous image
./cli/sdlc/cloud-run-grpc/deploy.sh dev --skip-build

# 3. For frontend rollback, redeploy previous version
cd web-ng-m3
firebase hosting:rollback --project=construction-code-expert-dev

Monitoring and Verification

Health Checks

TODO: Health endpoint implementation needs investigation. See GitHub Issue #222

# Check gRPC service health
curl https://construction-code-expert-dev-856365345080.us-central1.run.app/health

# Check ESPv2 gateway health
curl https://construction-code-expert-esp2-dev-6yieikr6ca-uc.a.run.app/health

Log Monitoring

# Get recent Cloud Run logs
./get-cloud-run-logs.sh --env=dev --minutes=30

# Query specific job logs
gcloud logging read "resource.type=cloud_run_job AND resource.labels.job_name=plan-ingestion-processor-dev" \
--project=construction-code-expert-dev --limit=50

Service Status

# Check Cloud Run service status
gcloud run services describe construction-code-expert-dev \
--region=us-central1 \
--project=construction-code-expert-dev

# Check Cloud Run job status
gcloud run jobs list --region=us-central1 --project=construction-code-expert-dev

# Check Firebase hosting status
firebase hosting:channel:list --project=construction-code-expert-dev

Troubleshooting

Common Issues

1. Docker Build Failures

  • Ensure Maven build completed successfully
  • Check that thin JAR and dependencies exist in target/
  • Verify Dockerfile paths are correct

2. Image Push Failures

  • Authenticate Docker: gcloud auth configure-docker us-central1-docker.pkg.dev
  • Verify Artifact Registry permissions
  • Check network connectivity

3. Deployment Timeouts

  • Increase timeout in deployment script
  • Check Cloud Run service logs for errors
  • Verify service account has required permissions

4. Frontend Build Failures

  • Regenerate gRPC-Web sources
  • Clear node_modules and reinstall: rm -rf node_modules && npm install
  • Check environment variables are set correctly

Getting Help

# View help for any deployment script
./cli/sdlc/full-stack-deploy.sh --help
./cli/sdlc/cloud-run-grpc/deploy.sh --help
./cli/sdlc/cloud-run-job/deploy.sh --help