FK of relation "inputprompt__user" does not exist

[yiweig]

Hello! I just recently pulled Onyx and have been trying to get it up on my local machine via docker compose (Mac M1 Pro, 16GB ram), and I seem to be running into an issue with api_server when it starts while alembic is running the initial db migrations.

This is a completely new install from scratch, I've never run Onyx before.

Alembic seems to be complaining about a missing FK on the inputprompt__user table.

Looking at the logs from api_server, it seems to start okay:

...
2025-01-31 10:44:55 INFO  [alembic.runtime.migration] Running upgrade  -> 47433d30de82, Create IndexAttempt table
2025-01-31 10:44:55 INFO  [alembic.runtime.migration] Running upgrade 47433d30de82 -> 6d387b3196c2, Basic Auth
2025-01-31 10:44:55 INFO  [alembic.runtime.migration] Running upgrade 6d387b3196c2 -> 2666d766cb9b, Google OAuth2
...

But then it gets to migration 33ea50e88f24 (which I see was recently merged in #3836) and it seems to stumble:

2025-01-31 10:45:12 INFO  [alembic.runtime.migration] Running upgrade f1ca58b2f2ec -> 4d58345da04a, lowercase_user_emails
2025-01-31 10:45:12 INFO  [alembic.runtime.migration] Running upgrade 4d58345da04a -> a6df6b88ef81, remove recent assistants
2025-01-31 10:45:12 INFO  [alembic.runtime.migration] Running upgrade a6df6b88ef81 -> 33ea50e88f24, foreign key input prompts
2025-01-31 10:45:12 ERROR [env_py] Error migrating schema public: (sqlalchemy.dialects.postgresql.asyncpg.ProgrammingError) <class 'asyncpg.exceptions.UndefinedTableError'>: relation ""user"" does not exist
2025-01-31 10:45:12 [SQL: ALTER TABLE inputprompt__user ADD CONSTRAINT inputprompt__user_user_id_fkey FOREIGN KEY(user_id) REFERENCES """user""" (id) ON DELETE CASCADE]
2025-01-31 10:45:12 (Background on this error at: https://sqlalche.me/e/20/f405)
2025-01-31 10:45:12 Traceback (most recent call last):
2025-01-31 10:45:12   File "/usr/local/lib/python3.11/site-packages/sqlalchemy/dialects/postgresql/asyncpg.py", line 530, in _prepare_and_execute
2025-01-31 10:45:12     self._rows = await prepared_stmt.fetch(*parameters)
2025-01-31 10:45:12                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-31 10:45:12   File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 176, in fetch
2025-01-31 10:45:12     data = await self.__bind_execute(args, 0, timeout)
2025-01-31 10:45:12            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-31 10:45:12   File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 241, in __bind_execute
2025-01-31 10:45:12     data, status, _ = await self.__do_execute(
2025-01-31 10:45:12                       ^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-31 10:45:12   File "/usr/local/lib/python3.11/site-packages/asyncpg/prepared_stmt.py", line 230, in __do_execute
2025-01-31 10:45:12     return await executor(protocol)
2025-01-31 10:45:12            ^^^^^^^^^^^^^^^^^^^^^^^^
2025-01-31 10:45:12   File "asyncpg/protocol/protocol.pyx", line 201, in bind_execute
2025-01-31 10:45:12 asyncpg.exceptions.UndefinedTableError: relation ""user"" does not exist

This causes api_server to attempt a restart, and then the error changes (maybe the user relation is being created by a separate db migration branch that catches up separately? I couldn't find evidence of this, but frankly I could have missed this in the logs). Anyways, the error then changes to

2025-01-31 10:45:12 sqlalchemy.exc.ProgrammingError: (sqlalchemy.dialects.postgresql.asyncpg.ProgrammingError) <class 'asyncpg.exceptions.UndefinedTableError'>: relation ""user"" does not exist
2025-01-31 10:45:12 [SQL: ALTER TABLE inputprompt__user ADD CONSTRAINT inputprompt__user_user_id_fkey FOREIGN KEY(user_id) REFERENCES """user""" (id) ON DELETE CASCADE]
2025-01-31 10:45:12 (Background on this error at: https://sqlalche.me/e/20/f405)
2025-01-31 10:45:14 INFO  [alembic.runtime.migration] Context impl PostgresqlImpl.
2025-01-31 10:45:14 INFO  [alembic.runtime.migration] Will assume transactional DDL.
2025-01-31 10:45:17 None of PyTorch, TensorFlow >= 2.0, or Flax have been found. Models won't be available and only tokenizers, configuration and file/data utilities can be used.
2025-01-31 10:45:17 INFO  [alembic.runtime.migration] Running upgrade a6df6b88ef81 -> 33ea50e88f24, foreign key input prompts
2025-01-31 10:45:17 ERROR [env_py] Error migrating schema public: (sqlalchemy.dialects.postgresql.asyncpg.ProgrammingError) <class 'asyncpg.exceptions.UndefinedObjectError'>: constraint "inputprompt__user_user_id_fkey" of relation "inputprompt__user" does not exist
2025-01-31 10:45:17 [SQL: ALTER TABLE inputprompt__user DROP CONSTRAINT inputprompt__user_user_id_fkey]

And then api_server becomes stuck in a boot loop with this error.

This was the last successful migration:

[postgres] # select * from alembic_version;
 version_num
--------------
 a6df6b88ef81
(1 row)

I can't seem to find the FK constraint that migration 33ea50e88f24 is attempting to drop:

[postgres] # \d inputprompt__user
              Table "public.inputprompt__user"
     Column      |  Type   | Collation | Nullable | Default
-----------------+---------+-----------+----------+---------
 input_prompt_id | integer |           | not null |
 user_id         | uuid    |           | not null |
 disabled        | boolean |           | not null |
Indexes:
    "inputprompt__user_pkey" PRIMARY KEY, btree (input_prompt_id, user_id)

[postgres] #

Any tips? I've tried completely removing all containers and volumes and repulling/rebuilding all services from scratch, but no luck. Happy to provide more logs if needed.

[potchy]

Same thing here :(

[jgable01]

Having the same issue here. Have tried destroying the database volume multiple times as well to trigger a fresh boot. Seems like they're working on a fix, if the commit history is anything to go off.

[yiweig]

Ah yeah @jgable01 seems like a fix just got merged in #3865, looks like we'll just have to wait!