bug: Schema not found the second time a spark table is accessed

[christophediprima]

What happened?

I am trying to insert into two different iceberg tables using spark connect. But as soon as I access a table using get_schema,table or insert I won't be able to do it a second time.

My code looks like this:

import ibis
from   pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()
con = ibis.pyspark.connect(session)

new_dataframe = Dataframe creation here

con.insert(
  "my_table",
  new_dataframe,
  database="my_catalog.my_database",
)

new_dataframe_2 = Dataframe creation here

con.insert(
  "my_other_table",
  new_dataframe_2,
  database="my_catalog.my_database",
)

First insert will work but not the second one. If first insert is commented, second one will work. Same thing will happen regardless of the get_schema,table or insert method.

What version of ibis are you using?

ibis-framework[pyspark]==10.0.0.dev490

What backend(s) are you using, if any?

PySpark

Relevant log output

pyspark.errors.exceptions.connect.AnalysisException: [SCHEMA_NOT_FOUND] The schema `my_catalog`. cannot be found. Verify the spelling and correctness of the schema and catalog.
If you did not qualify the name with a catalog, verify the current_schema() output, or qualify the name with the correct catalog.
To tolerate the error on drop use DROP SCHEMA IF EXISTS.

Code of Conduct

• I agree to follow this project's Code of Conduct

[cpcloud]

Thanks for the issue!

Can you show a fully reproducible example that I can copy-paste into a file or a Python REPL?

The

Dataframe creation here

bits seem critical to reproducing the problem for example.

Toy data is completely fine.

[christophediprima]

Sure ! Please let me know if this one is good enough:

import ibis
from   pyspark.sql import SparkSession

session = SparkSession.builder.getOrCreate()
con = ibis.pyspark.connect(session)

# Could not use following as 'force' is not supported in 'create_table' and does not seem to work in 'create_database'
# con.create_database(
#   "my_database",
#   force=True,
# )
# con.create_table(
#   "my_table",
#   schema=ibis.schema([("id", "string")]),
#   database="my_database",
#   force=True,
# )

con.sql(f"CREATE DATABASE IF NOT EXISTS my_database")
con.sql("""
  CREATE TABLE IF NOT EXISTS my_database.my_table (
    id string
  )
""")

new_ids = ibis.memtable([{
  'id': 'my_id',
  }])

con.insert(
  "my_table",
  new_ids,
  database="my_database",
)

new_ids_2 = ibis.memtable([{
  'id': 'my_id_2',
  }])

con.insert(
  "my_table",
  new_ids_2,
  database="my_database",
)

[gforsyth]

Hey @christophediprima !

The .sql method is for SQL queries that return results that you want to chain inline with other Ibis commands. It is not a good idea to try using it for DDL operations. If you need to do some DDL outside of Ibis' API, you should use con.raw_sql for that instead.

[gforsyth]

There does seem to be a bug here, which is that force=True on con.create_database isn't generating the expected IF NOT EXISTS clause

[gforsyth]

Found the issue, I'll put up a PR

[christophediprima]

Thanks for your replies. I used .sql as a workaround so I can provide a working full example to reproduce the initial bug I was reporting.

Did you find the solution for my initial bug report or force=True one?

[gforsyth]

force=True -- I can't reproduce your initial bug if the database and table have been created

[gforsyth]

(Although that's with a local spark session, not using Iceberg)