8000
Skip to content

Database readiness checks do not detect authentication failures #2458

Description

@junhaoliao

Bug

The bundled database health checks use mysqladmin ping with CLP's application credentials:

Both MariaDB and MySQL document that mysqladmin ping exits with status 0 when the server is running even if authentication fails with Access denied: MariaDB documentation, MySQL documentation.

Consequently:

  • Docker Compose can mark the database healthy and start dependent services even though they cannot authenticate.
  • Kubernetes can mark the database pod ready despite invalid CLP credentials.
  • The Helm liveness and readiness probes currently test the same process-availability condition rather than separate liveness and application readiness.
  • Credential drift is particularly confusing with persistent database data because changing container initialization variables does not update accounts in an existing database.

Expected behavior:

  • Docker Compose health should fail until the configured CLP database, username, and password are usable.
  • Helm readiness should perform a lightweight authenticated operation, such as connecting to the configured database and executing SELECT 1.
  • Helm liveness should remain a process-level check so an application credential error does not cause unnecessary database restarts.
  • The implementation should support both configured database types: MariaDB and MySQL. A MariaDB-specific healthcheck.sh command should not be used unconditionally in the shared Helm template.
  • Tests should cover both database types and verify that readiness fails with invalid credentials.

CLP version

main at 979b93c4e0e6d0edbc1cf933c02c6eedc1cdf148

Environment

This issue is deployment-environment independent and affects:

  • Docker Compose package deployments
  • Helm/Kubernetes package deployments
  • Bundled MariaDB (mariadb:10.11.16, the default)
  • Bundled MySQL (mysql:8.0.46, supported by Helm)

Reproduction steps

  1. Start a bundled database using the Docker Compose or Helm deployment.

  2. Run mysqladmin ping against the running database with an invalid password:

    mysqladmin ping \
      --host=127.0.0.1 \
      --port=3306 \
      --user=clp-user \
      --password=incorrect
    echo "$?"
  3. Observe an Access denied error but an exit status of 0.

  4. In Docker Compose, observe that this result satisfies the database health check.

  5. In Kubernetes, observe that the same result satisfies both the database readiness and liveness probes.

  6. Attempt to start or use a CLP service with the same invalid credentials and observe that it cannot connect to the database.

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions

    0