fix(ENGKNOW-3691): accept db credentials from the host application

VERSION

@@ -1 +1 @@
-5.11.1
+5.11.2

documentation/design/datbase.md

@@ -23,11 +23,13 @@ These come in two flavors:
 
 Notes:
 1. Uses DBNorIterator.
+2. Resolves against the **user** connections, i.e. the `gor.db.credentials` file. See [Configuration](#configuration).
 
 #### **sql:// URIs**
 
 Notes:
 1. Uses DBNorIterator (and SQLSource).
+2. Resolves against the **user** connections, i.e. the `gor.db.credentials` file. See [Configuration](#configuration).
 
 ### **Limited SQL Commands**
 These come in  flavors:
@@ -50,6 +52,7 @@ gor db://rda:variant_annotations | top 10
 
 Notes:
 1. Uses DBSource and DbGenomicIterator.
+2. Resolves against the **system** connections, i.e. the credentials the host passes in. See [Configuration](#configuration).
 
 #### **//db/ Paths**
 The //db/ paths are arbitrary SELECT statements that can be used to selected from a given database.  
@@ -68,7 +71,8 @@ Their limit is the can ONLY be executed from a link file  but not from a GOR que
 
 Notes:  
 1. Uses DBNorIterator.
-2. This is the old style of doing SQL access in GOR.  The new preferred way is to use the `sql`, `norsql`, `gorsql` commands.  This is likely to be deprecated.
+2. Resolves against the **system** connections, i.e. the credentials the host passes in. See [Configuration](#configuration).
+3. This is the old style of doing SQL access in GOR.  The new preferred way is to use the `sql`, `norsql`, `gorsql` commands.  This is likely to be deprecated.
 
 ### Sepcial Variables
 
@@ -79,23 +83,58 @@ TBD
 
 ### Configuration
 
-TBD
+There are two connection caches, and in a server they are fed from **separate** sources:
+
+| Cache | Used by | Fed from |
+|---|---|---|
+| system connections | internal use (e.g. session management) and the access-controlled operations `db://`, `//db:` | credentials the host passes in |
+| user connections | the user-available commands and sources `SQL`, `GORSQL`, `NORSQL`, `sql://` | the `gor.db.credentials` file |
+
+They are kept apart deliberately. System credentials typically rotate, and a file cannot hold rotating
+credentials without going stale; equally, the rotating system credentials are not the ones user queries
+should reach.
+
+In a console app there is no host to supply credentials, so both caches load from the file.
+
+#### The credentials file
+
+`gor.db.credentials` is tab-separated, with a header line:
+
+```
+name	driver	url	user	pwd
+rda	org.postgresql.Driver	jdbc:postgresql://myurl.com:5432/csa	rda	mypass
+```
+
+The password column is optional. Lines starting with `#` are ignored.
+
+Its location defaults to the config directory and can be set with the `gor.db.credentials` system
+property.
+
+A missing credentials file is an error.
+
+#### Credentials passed in by the host
+
+The host application supplies the system credentials as `DbCredentials`:
+
+```java
+var systemCredentials = List.of(new DbCredentials("rda", url, user, password));
+
+DbConnection.initInServer(systemCredentials);
+// which is, per cache:
+DbConnection.systemConnections.initializeDbSources(systemCredentials);
+DbConnection.userConnections.initializeDbSources(credpath);
+```
+
+Gor never reads credentials from the environment itself, and does not care where the host got them —
+its own configuration, a secret manager, or environment variables the host owns the naming of. This
+keeps deployment-specific naming out of the library.
+
+`DbCredentials` takes `name`, `url`, `user`, `pwd`, and an optional `driver`. When `driver` is null it
+is derived from the url prefix (`jdbc:postgresql:`, `jdbc:oracle:`). Blank values count as unset,
+including the password, so a secret manager rendering an empty string does not become a real empty
+password. Incomplete credentials are logged — naming the missing field, never a value — and skipped,
+rather than failing startup.
 
-We have two different configuration files:
-*
-* <li> gor.db.credentials - contains the system databases, which are used by the system internally (e.g. session management),
-*                         and by operation where we have strict access controls (db://, //db:).  These credentials
-*                         typically would grant full access to the database.
-* <li> gor.sql.credentials - contains the user databases, which are used by the user available commands/sources (SQL,
-*                          GORSQL, NORSQL, sql://).  These credentials typically would grant limited/read-only access to
-*                          the database.
-* <p><br>
-* The format for these files is:
-* <pre>
-*    name\tdriver\turl\tuser\tpwd
-*    rda\torg.postgresql.Driver\tjdbc:postgresql://myurl.com:5432/csa\trda\tmypass
-*    ...
-* </pre>
-*<p>
-* The location of the files defaults to the config directory but can be specified by the system properties
-* gor.db.credentials and gor.sql.credentials.
+The two `initializeDbSources` overloads are alternatives, not additive: each clears the cache first.
+That is what keeps a cache fed from exactly one source. Passing no credentials therefore leaves the
+system cache empty, and `db://` sources will not resolve.

documentation/src/command/GORSQL.rst

@@ -7,7 +7,7 @@
 ======
 GORSQL
 ======
-The :ref:`GORSQL` command allows you to run arbitrary SQL commands against "the database" (the database here being defined by the content of a file called gor.sql.credentials in the config directory).
+The :ref:`GORSQL` command allows you to run arbitrary SQL commands against "the database" (the databases available here are the user connections, defined by the content of a file called gor.db.credentials in the config directory).
 
 For :ref:`GORSQL` to work properly, the defined query must return Chrom-POS information as first two columns.
 
@@ -33,7 +33,7 @@ Options
 | ``-ff File``         | Read tags from a tag file and filter files and file contents on. Also accepts a nested query.     |
 |                      | The following place holders are provided: #{TAGS}.   The necessary quoting is done automatically. |
 +----------------------+---------------------------------------------------------------------------------------------------+
-| ``-db database``     | Database alias as defied in ``gor.sql.credentials``.                                              |
+| ``-db database``     | Database alias as defied in ``gor.db.credentials``.                                               |
 +----------------------+---------------------------------------------------------------------------------------------------+
 
 

documentation/src/command/NORSQL.rst

@@ -7,7 +7,7 @@
 ======
 NORSQL
 ======
-The :ref:`NORSQL` command allows you to run arbitrary SQL commands against "the database" (the database here being defined by the content of a file called gor.sql.credentials in the config directory).
+The :ref:`NORSQL` command allows you to run arbitrary SQL commands against "the database" (the databases available here are the user connections, defined by the content of a file called gor.db.credentials in the config directory).
 
 :ref:`NORSQL` can be run against any database table.
 

documentation/src/command/SQL.rst

@@ -7,7 +7,7 @@
 ===
 SQL
 ===
-The SQL command allows you to run arbitrary SQL commands against "the database" (the database here being defined by the content of a file called gor.sql.credentials in the config directory).
+The SQL command allows you to run arbitrary SQL commands against "the database" (the databases available here are the user connections, defined by the content of a file called gor.db.credentials in the config directory).
 
 SQL statements need to be encapsulated between curly brackets, e.g. sql {sql_query}.
 
@@ -37,7 +37,7 @@ Options
 | ``-ff File``         | Read tags from a tag file and filter files and file contents on. Also accepts a nested query.     |
 |                      | The following place holders are provided: #{TAGS}.   The necessary quoting is done automatically. |
 +----------------------+---------------------------------------------------------------------------------------------------+
-| ``-db database``     | Database alias as defied in ``gor.sql.credentials``.                                              |
+| ``-db database``     | Database alias as defied in ``gor.db.credentials``.                                               |
 +----------------------+---------------------------------------------------------------------------------------------------+
 
 

gortools/src/test/java/gorsat/UTestSQLInputSource.java

@@ -58,9 +58,9 @@ public static void initDb() throws IOException, ClassNotFoundException, SQLExcep
 
         List<String> credFileLines = Files.readAllLines(Path.of(rdaPaths[2]));
         credFileLines.add(Files.readAllLines(Path.of(avasPaths[2])).get(1));
-        File credFile = FileTestUtils.createTempFile(new File(rdaPaths[0]), "gor.sql.credentials",
+        File credFile = FileTestUtils.createTempFile(new File(rdaPaths[0]), "gor.db.credentials",
                 credFileLines.stream().collect(Collectors.joining("\n")));
-        System.setProperty("gor.sql.credentials", credFile.getAbsolutePath());
+        System.setProperty("gor.db.credentials", credFile.getAbsolutePath());
         DbConnection.initInConsoleApp();
     }
 

gortools/src/test/scala/gorsat/UTestInputSourceParsing.scala

@@ -119,7 +119,7 @@ class UTestInputSourceParsing extends AnyFunSuite with BeforeAndAfter with Mocki
 
     Class.forName("org.apache.derby.jdbc.EmbeddedDriver")
     val paths = createTestDataBase_Derby
-    System.setProperty("gor.sql.credentials", paths(2))
+    System.setProperty("gor.db.credentials", paths(2))
     DbConnection.initInConsoleApp()
 
     var tempDirectory = FileTestUtils.createTempDirectory(this.getClass.getName)

model/src/main/java/org/gorpipe/gor/model/DbConnection.java

@@ -34,6 +34,7 @@
 import java.io.IOException;
 import java.lang.reflect.UndeclaredThrowableException;
 import java.sql.*;
+import java.util.List;
 
 /**
  * DbConnection abstract access to database source that are configured on installation time.
@@ -43,24 +44,30 @@
  * The static part creates a cache of DbConnection objects that can be used to access all databases defined in the
  * database configuration files.  This part is located here for backward compatibility reasons.
  *<p>
- * We have two different configuration files:
+ * We have one configuration file:
  *
- * <li> gor.db.credentials - contains the system databases, which are used by the system internally (e.g. session management),
- *                         and by operation where we have strict access controls (db://, //db:).  These credentials
- *                         typically would grant full access to the database.
- * <li> gor.sql.credentials - contains the user databases, which are used by the user available commands/sources (SQL,
- *                          GORSQL, NORSQL, sql://).  These credentials typically would grant limited/read-only access to
- *                          the database.
+ * <li> gor.db.credentials - contains all the databases gor can reach.  It feeds both the system connections,
+ *                         used internally (e.g. session management) and by operations with strict access
+ *                         controls (db://, //db:), and the user connections behind the user available
+ *                         commands/sources (SQL, GORSQL, NORSQL, sql://).
  * <p><br>
- * The format for these files is:
+ * In a server, the two caches are fed from separate sources.  {@link #userConnections} comes from the
+ * gor.db.credentials file, while {@link #systemConnections} comes from credentials the host passes in
+ * as {@link DbCredentials} — see {@link #initInServer(java.util.List)}.  System credentials typically
+ * rotate, and so cannot be baked into a file without going stale; gor does not care where the host got
+ * them and never reads them from the environment itself.
+ * <p><br>
+ * In a console app there is no host to supply credentials, so both caches load from the file.
+ * <p><br>
+ * The format for this file is:
  * <pre>
  *    name\tdriver\turl\tuser\tpwd
  *    rda\torg.postgresql.Driver\tjdbc:postgresql://myurl.com:5432/csa\trda\tmypass
  *    ...
  * </pre>
  *<p>
- * The location of the files defaults to the config directory but can be specified by the system properties
- * gor.db.credentials and gor.sql.credentials.
+ * The location of the file defaults to the config directory but can be specified by the system property
+ * gor.db.credentials.
  * <p><br>
  * TODO:  This class needs some refactoring to handle the different databases more gracefully.  We should use some kind of
  * plugin mechanism (or Guice) instead of if/else statements.
@@ -244,10 +251,7 @@ public static void initInConsoleApp() throws ClassNotFoundException, IOException
         File homeDbCredFile = new File(System.getProperty("user.home"), "gor.db.credentials");
         final String dbCredpath = homeDbCredFile.exists() ? homeDbCredFile.getCanonicalPath() : System.getProperty("gor.db.credentials");
         systemConnections.initializeDbSources(dbCredpath);
-
-        File homeSqlCredFile = new File(System.getProperty("user.home"), "gor.sql.credentials");
-        final String sqlCredpath = homeSqlCredFile.exists() ? homeSqlCredFile.getCanonicalPath() : System.getProperty("gor.sql.credentials");
-        userConnections.initializeDbSources(sqlCredpath);
+        userConnections.initializeDbSources(dbCredpath);
 
         setPasswordCallback(s -> {
             final Console console = System.console();
@@ -261,16 +265,39 @@ public static void initInConsoleApp() throws ClassNotFoundException, IOException
      * @throws ClassNotFoundException
      * @throws IOException
      */
-    @SuppressWarnings("unused") // Called from GorQueryTask in gor-services
+    @SuppressWarnings("unused") // Called from gor-services
     public static void initInServer() throws ClassNotFoundException, IOException {
-        final String dbCredpath = System.getProperty("gor.db.credentials");
-        if (dbCredpath != null) {
-            systemConnections.initializeDbSources(dbCredpath);
-        }
+        initInServer(List.of());
+    }
+
+    /**
+     * Initialize DbSource to be used in a server, with the system credentials supplied by the host
+     * application.
+     *
+     * The two caches are fed from separate sources:
+     *
+     * <li> {@link #systemConnections}, used internally and by the access-controlled operations
+     *      (db://, //db:), comes from the supplied credentials. These typically rotate, so a host
+     *      holding them — in its own configuration, a secret manager, or environment variables it
+     *      owns the naming of — passes them here rather than gor reading them itself.
+     * <li> {@link #userConnections}, behind the user-available commands and sources (SQL, GORSQL,
+     *      NORSQL, sql://), comes from the gor.db.credentials file.
+     *
+     * The sources are kept apart deliberately: the file cannot hold credentials that rotate without
+     * going stale, and the rotating system credentials are not the ones user queries should reach.
+     *
+     * @param systemCredentials Credentials for {@link #systemConnections}. May be empty, which leaves
+     *                          that cache empty.
+     * @throws ClassNotFoundException
+     * @throws IOException
+     */
+    @SuppressWarnings("unused") // Called from GorQueryTask in gor-services
+    public static void initInServer(List<DbCredentials> systemCredentials) throws ClassNotFoundException, IOException {
+        systemConnections.initializeDbSources(systemCredentials);
 
-        final String sqlCredpath = System.getProperty("gor.sql.credentials");
-        if (sqlCredpath != null) {
-            userConnections.initializeDbSources(sqlCredpath);
+        final String userCredpath = System.getProperty("gor.db.credentials");
+        if (userCredpath != null) {
+            userConnections.initializeDbSources(userCredpath);
         }
     }