Skip to main content

UI Testing with Cypress

This document provides comprehensive guidance on end-to-end (E2E) testing of the PermitProof web application using Cypress.

Overview​

The project uses Cypress for automated end-to-end testing of the Angular Material 3 frontend (web-ng-m3). The test suite includes automated Firebase authentication without manual token management, enabling comprehensive testing of authenticated user workflows.

Key Achievement​

🎯 Programmatic Authentication: We've successfully implemented automated authentication for Cypress E2E tests, enabling full-stack UI testing without manual login or token management.

Available Test Suites​

TODO: Many tests currently have hardcoded project IDs and assume specific project states. These need to be refactored to use a baseline project pattern for idempotency and CI/CD compatibility. See GitHub Issue #223 for implementation details.

The Cypress test suite is located in web-ng-m3/cypress/e2e/ and includes the following test categories:

Authentication Tests​

Project Management Tests​

Feature-Specific Tests​

Admin & UI Tests​

End-to-End Test Scenarios​

Implemented Scenarios​

  1. Complete Project Workflow

    • User authentication
    • Project creation
    • PDF upload and ingestion
    • Code applicability analysis
    • Compliance report generation
    • Project deletion with safety confirmations

  2. Billing Operations

    • View account balance
    • Check billing profile
    • Verify transaction history
    • Test gRPC billing service calls

  3. Background Task Monitoring

    • View task progress
    • Check task status updates
    • Verify real-time progress tracking

  4. Project Management

    • Create new projects
    • Copy existing projects
    • Delete projects (soft and hard delete)
    • Manage project files

Potential Future Scenarios​

The following scenarios can be implemented using the existing Cypress infrastructure:

  1. Multi-User Collaboration

    • Share projects with other users
    • Test different permission levels (Owner, Editor, Viewer)
    • Verify access control enforcement

  2. Code Analysis Workflows

    • Select building codes for analysis
    • Configure analysis parameters
    • Review code applicability results
    • Generate and download compliance reports

  3. User Profile Management

    • Update user settings
    • Manage notification preferences
    • View usage statistics

  4. Mobile Responsiveness

    • Test responsive layouts
    • Verify mobile navigation
    • Check touch interactions

  5. Error Handling

    • Test network failure scenarios
    • Verify error message display
    • Check retry mechanisms

  6. Performance Testing

    • Measure page load times
    • Check for memory leaks
    • Verify lazy loading behavior

Codifying Manual Verification​

When developing new features, we often perform manual verification steps. It is highly recommended to codify these steps into Cypress tests to ensure long-term stability and prevent regressions.

Example: Chat Feature Verification​

Currently, the Chat feature is verified manually. Here is how we can codify this process into a Cypress test:

Manual Steps:

  1. Log in to the application.
  2. Navigate to a project.
  3. Open the chat drawer.
  4. Verify the welcome message and assistant avatar.
  5. Send a message (e.g., "Hello").
  6. Verify the message appears in the chat history.
  7. Verify the assistant responds.

Codified Cypress Test (Proposed):

describe('Chat Feature', () => {
  const testUser = 'ai-swe-agent-test@codetricks.org';
  const projectId = 'construction-code-expert-test';

  beforeEach(() => {
    cy.loginByFirebase(testUser, projectId);
    cy.wait(3000);
    // Navigate to a specific project (assuming one exists or create one)
    cy.contains('ProjectName').click(); 
  });

  it('should open chat and send a message', () => {
    // Open Chat Drawer
    cy.get('[data-testid="chat-toggle-button"]').click();
    cy.get('.chat-drawer').should('be.visible');

    // Verify Welcome State
    cy.get('.empty-state-logo').should('be.visible');
    cy.contains("Hi! I'm your PermitProof assistant.").should('be.visible');

    // Send Message
    cy.get('textarea[placeholder="Ask a question..."]').type('Hello{enter}');

    // Verify User Message
    cy.get('.user-message').contains('Hello').should('be.visible');

    // Verify Assistant Response (wait for it)
    cy.get('.assistant-message', { timeout: 10000 }).should('exist');
    cy.get('.assistant-message .avatar img').should('have.attr', 'alt', 'PermitProof Assistant');
  });
});

Benefits of Codification:

  • Reproducibility: Tests run exactly the same way every time.
  • Regression Testing: Ensures new changes don't break existing chat functionality.
  • Documentation: The test itself serves as documentation for how the feature should behave.

Antigravity IDE Workflow​

For developers using the Antigravity IDE with Browser Use Capability, the workflow is streamlined:

Browser Authentication​

The Antigravity IDE initializes the browser with a custom Chrome Profile that is already authenticated with the Google Account ai-swe-agent-test@codetricks.org.

  • This account is allowlisted to access the test environment.
  • It has been granted access to select Architectural Projects in the test environment.
  • No manual token generation is required when using the IDE's browser tool.
  1. Reproduce Manually: Open the browser using the IDE tool, reproduce the issue manually, and take a screenshot.
  2. Codify Failure: Create a Cypress test that reproduces the issue and validate that it fails.
  3. Fix & Deploy: Implement the fix, build the application, and deploy to the test environment.
  4. Verify (Automated): Rerun the Cypress test and validate that it passes.
  5. Verify (Manual): Rerun the manual test in the browser and validate visually.
  6. Merge: Create a pull request to merge the changes.

Running Cypress Tests​

Prerequisites​

  1. Environment Setup: Ensure the test environment is properly configured
  2. Frontend Deployment: Deploy the frontend to the test environment with authentication modifications
  3. Test User: Verify ai-swe-agent-test@codetricks.org exists in Firebase with appropriate permissions
  4. Service Account: Ensure the AI SWE Agent service account has Firebase Admin privileges

Local Development Testing​

# Navigate to the Angular application directory
cd web-ng-m3

# Install dependencies (if not already done)
npm install

# Start the local development server
npm run start:test

# In a separate terminal, open Cypress interactive mode
npx cypress open

# Or run all tests in headless mode
npx cypress run --browser chromium

Testing Against Deployed Environment​

# Navigate to the Angular application directory
cd web-ng-m3

# Run tests against the deployed test environment
npx cypress run \
  --spec "cypress/e2e/**/*.cy.ts" \
  --browser chromium \
  --config baseUrl=https://construction-code-expert-test-m3.web.app

Running Specific Test Suites​

# Run only authentication tests
npx cypress run --spec "cypress/e2e/auth-*.cy.ts"

# Run only project management tests
npx cypress run --spec "cypress/e2e/project-*.cy.ts"

# Run only billing tests
npx cypress run --spec "cypress/e2e/billing-*.cy.ts"

# Run a single test file
npx cypress run --spec "cypress/e2e/project-delete-test.cy.ts"

Authentication in Cypress Tests​

How Authentication Works​

The Cypress test suite uses a sophisticated authentication mechanism that integrates with Firebase:

  1. Application Detects Cypress: The Angular app checks for window.Cypress and skips automatic redirects during tests
  2. Token Generation: Cypress automatically invokes the ./firebase-token-generator/generate-token.sh script for each test
  3. Programmatic Login: The app's cypressLogin() method accepts ID tokens and establishes authenticated sessions
  4. Preserved Auth State: Tests navigate using clicks rather than page reloads to maintain authentication

Custom Cypress Commands​

The test suite includes a custom loginByFirebase command defined in cypress/support/commands.ts:

Cypress.Commands.add('loginByFirebase', (email: string, projectId: string) => {
  return cy.task('generateToken', { email, projectId }).then((customToken) => {
    return cy.window().then((win: any) => {
      return new Cypress.Promise((resolve, reject) => {
        // Wait for cypressLogin method to be available
        const checkForFirebase = () => {
          if (win.cypressLogin) {
            win.cypressLogin(customToken as string)
              .then((userCredential: any) => {
                resolve(userCredential);
              })
              .catch((err: any) => {
                reject(err);
              });
          } else {
            // Retry until timeout
            setTimeout(checkForFirebase, 200);
          }
        };
        checkForFirebase();
      });
    });
  });
});

Token Generation Configuration​

The cypress.config.js file defines a custom task for generating Firebase tokens:

module.exports = defineConfig({
  e2e: {
    baseUrl: 'http://localhost:4200',
    setupNodeEvents(on, config) {
      on('task', {
        generateToken({ email, projectId }) {
          const command = `
            export GOOGLE_APPLICATION_CREDENTIALS=$(pwd)/.secrets/agent-credentials/${projectId}.ai-swe-agent.json && \\
            source env/${projectId.split('-').pop()}/firebase/m3/setvars.secrets.sh && \\
            ./firebase-token-generator/generate-token.sh ${email}
          `;
          const token = execSync(command, { shell: '/bin/bash', cwd: '../' }).toString().trim();
          return token;
        }
      });
    }
  }
});

Example Test Pattern with Authentication​

describe('Authenticated Feature Test', () => {
  const testUser = 'ai-swe-agent-test@codetricks.org';
  const projectId = 'construction-code-expert-test';

  beforeEach(() => {
    // Clear any existing authentication state
    cy.clearCookies();
    cy.clearLocalStorage();
    cy.window().then((win) => {
      win.sessionStorage.clear();
    });
  });

  it('should test authenticated functionality', () => {
    // Step 1: Visit the application
    cy.visit('/');
    
    // Step 2: Wait for initial loading to complete
    cy.get('body').should('not.contain.text', 'Initializing Application', { timeout: 30000 });
    
    // Step 3: Login with Firebase
    cy.loginByFirebase(testUser, projectId);
    
    // Step 4: Wait for authentication to propagate
    cy.wait(3000);
    
    // Step 5: Navigate using clicks (preserves auth state)
    cy.contains('ProjectName').click();
    
    // Step 6: Verify authenticated content is visible
    cy.contains('Protected Content').should('be.visible');
  });
});

Critical Authentication Notes​

  1. Timing: Always add cy.wait(3000) after cy.loginByFirebase() to allow authentication state to propagate through the application

  2. Navigation Strategy: Use cy.contains().click() instead of cy.visit() after authentication to preserve session state

  3. Test User Permissions: The ai-swe-agent-test@codetricks.org user must have appropriate permissions (typically OWNER role) for the test scenarios

  4. Environment Variables: Ensure the test environment has the correct Firebase configuration and service account credentials

CI/CD Integration​

GitHub Actions Workflow​

Cypress tests can be integrated into GitHub Actions for continuous testing:

name: E2E Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main, develop ]

jobs:
  cypress-tests:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v3
      
      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
      
      - name: Install dependencies
        working-directory: web-ng-m3
        run: npm ci
      
      - name: Setup Firebase credentials
        env:
          FIREBASE_CREDENTIALS: ${{ secrets.FIREBASE_CREDENTIALS }}
        run: |
          mkdir -p .secrets/agent-credentials
          echo "$FIREBASE_CREDENTIALS" > .secrets/agent-credentials/construction-code-expert-test.ai-swe-agent.json
      
      - name: Run Cypress tests
        working-directory: web-ng-m3
        run: |
          npx cypress run \
            --browser chromium \
            --config baseUrl=https://construction-code-expert-test-m3.web.app
      
      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v3
        with:
          name: cypress-results
          path: web-ng-m3/cypress/screenshots

Cloud Build Integration​

For Google Cloud Build, add a step to your cloudbuild.yaml:

steps:
  # ... other build steps ...
  
  - name: 'cypress/included:13.6.0'
    id: 'cypress-tests'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        cd web-ng-m3
        npm ci
        npx cypress run --browser chromium --config baseUrl=https://construction-code-expert-test-m3.web.app
    env:
      - 'GOOGLE_APPLICATION_CREDENTIALS=/workspace/.secrets/agent-credentials/construction-code-expert-test.ai-swe-agent.json'

Common Gotchas and Troubleshooting​

1. Authentication Timeout​

Problem: Tests fail with "Timed out waiting for window.firebase.auth to be available"

Solution:

  • Ensure the frontend is deployed with the Cypress-aware authentication modifications
  • Check that window.cypressLogin method is exposed in the application
  • Verify the GOOGLE_APPLICATION_CREDENTIALS path is correct
  • Increase the timeout in cypress/support/commands.ts if needed

2. Auth State Not Persisting​

Problem: Tests lose authentication after navigation

Solution:

  • Use cy.contains().click() for navigation instead of cy.visit()
  • Avoid cy.reload() after authentication
  • Ensure cy.wait(3000) is called after cy.loginByFirebase()

3. Token Generation Fails​

Problem: generateToken task returns empty or invalid token

Solution:

  • Verify the service account JSON file exists at the expected path
  • Check that firebase-token-generator/generate-token.sh is executable
  • Ensure environment variables are sourced correctly
  • Verify the service account has Firebase Admin privileges

4. gRPC Call Interception Issues​

Problem: cy.intercept() doesn't capture gRPC calls

Solution:

  • gRPC-Web uses POST requests to specific endpoints
  • Use pattern matching: cy.intercept('POST', '**/ServiceName/MethodName').as('grpcCall')
  • Check the Network tab in Cypress to see actual request URLs
  • Ensure ESPv2 is properly configured for gRPC-Web transcoding

5. Element Not Found Errors​

Problem: Tests fail with "Element not found" even though element exists

Solution:

  • Add appropriate waits: cy.wait(2000) before interacting with elements
  • Use { timeout: 10000 } option for slow-loading elements
  • Check for overlays: cy.get('.cdk-overlay-backdrop').should('not.exist')
  • Scroll element into view: cy.contains('text').scrollIntoView()

6. Test Environment Isolation​

Problem: Tests interfere with each other or leave dirty state

Solution:

  • Use beforeEach() to clear cookies, localStorage, and sessionStorage
  • Create unique test data for each test run
  • Use soft delete instead of hard delete for cleanup
  • Consider using database snapshots for test isolation

7. Flaky Tests​

Problem: Tests pass sometimes and fail other times

Solution:

  • Add explicit waits instead of relying on implicit timeouts
  • Check for race conditions in async operations
  • Use cy.intercept() to wait for specific network requests
  • Increase timeouts for slow operations (e.g., LLM processing)

Best Practices​

  1. Use Descriptive Test Names: Make it clear what each test is verifying
  2. Keep Tests Independent: Each test should be able to run in isolation
  3. Use Custom Commands: Encapsulate common patterns in custom Cypress commands
  4. Add Logging: Use cy.task('printLog', message) for debugging
  5. Test Happy and Unhappy Paths: Verify both success and error scenarios
  6. Avoid Hard-Coded Delays: Use cy.wait('@alias') for network requests instead of cy.wait(5000)
  7. Clean Up Test Data: Ensure tests don't leave behind orphaned data
  8. Use Page Objects: For complex pages, consider using the Page Object pattern

External Resources​