BUG: single column DataFrame df.apply(f, raw=True, engine='numba') failed when f is a non-reduction function

[auderson]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

def f(x):
    return x

df1 = pd.DataFrame({"A": [1,2,3]})
df1.name = "df1"
df2 = pd.DataFrame({"A": [1,2,3], "B": [4,5,6]})
df2.name = "df2"

for df in [df1, df2]:
    for raw in [True, False]:
        for engine in ["python", "numba"]:
            try:
                df.apply(f, raw=raw, engine=engine)
            except Exception as e:
                print(f"{df.name}.apply(f, raw={raw}, engine={engine!r}) failed: {str(e)}")
            else:
                print(f"{df.name}.apply(f, raw={raw}, engine={engine!r}) passed")

Issue Description

The above code gives:

df1.apply(f, raw=True, engine='python') passed
df1.apply(f, raw=True, engine='numba') failed: Length of values (3) does not match length of index (1)
df1.apply(f, raw=False, engine='python') passed
df1.apply(f, raw=False, engine='numba') passed
df2.apply(f, raw=True, engine='python') passed
df2.apply(f, raw=True, engine='numba') passed
df2.apply(f, raw=False, engine='python') passed
df2.apply(f, raw=False, engine='numba') passed

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In[12], line 1
----> 1 df1.apply(f, raw=True, engine='numba')

File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/frame.py:10374, in DataFrame.apply(self, func, axis, raw, result_type, args, by_row, engine, engine_kwargs, **kwargs)
  10360 from pandas.core.apply import frame_apply
  10362 op = frame_apply(
  10363     self,
  10364     func=func,
   (...)
  10372     kwargs=kwargs,
  10373 )
> 10374 return op.apply().__finalize__(self, method="apply")

File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/apply.py:914, in FrameApply.apply(self)
    912 # raw
    913 elif self.raw:
--> 914     return self.apply_raw(engine=self.engine, engine_kwargs=self.engine_kwargs)
    916 return self.apply_standard()

File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/apply.py:1031, in FrameApply.apply_raw(self, engine, engine_kwargs)
   1029     return self.obj._constructor(result, index=self.index, columns=self.columns)
   1030 else:
-> 1031     return self.obj._constructor_sliced(result, index=self.agg_axis)

File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/series.py:575, in Series.__init__(self, data, index, dtype, name, copy, fastpath)
    573     index = default_index(len(data))
    574 elif is_list_like(data):
--> 575     com.require_length_match(data, index)
    577 # create/copy the manager
    578 if isinstance(data, (SingleBlockManager, SingleArrayManager)):

File ~/mambaforge/envs/py3.10/lib/python3.10/site-packages/pandas/core/common.py:573, in require_length_match(data, index)
    569 """
    570 Check the length of data matches the length of the index.
    571 """
    572 if len(data) != len(index):
--> 573     raise ValueError(
    574         "Length of values "
    575         f"({len(data)}) "
    576         "does not match length of index "
    577         f"({len(index)})"
    578     )

ValueError: Length of values (3) does not match length of index (1)

Expected Behavior

The issue happens on single column DataFrame with numba engine. Expected working as well as python engine.

Installed Versions

INSTALLED VERSIONS

commit : 0691c5c
python : 3.10.14
python-bits : 64
OS : Linux
OS-release : 5.15.0-122-generic
Version : #132-Ubuntu SMP Thu Aug 29 13:45:52 UTC 2024
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : en_US.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.2.3
numpy : 1.26.4
pytz : 2024.1
dateutil : 2.9.0
pip : 24.0
Cython : 3.0.7
sphinx : 7.3.7
IPython : 8.25.0
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.12.3
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : None
fsspec : 2024.6.0
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.4
lxml.etree : None
matplotlib : 3.9.2
numba : 0.60.0
numexpr : 2.10.0
odfpy : None
openpyxl : 3.1.5
pandas_gbq : None
psycopg2 : 2.9.9
pymysql : 1.4.6
pyarrow : 16.1.0
pyreadstat : None
pytest : 8.2.2
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.14.0
sqlalchemy : 2.0.31
tables : 3.9.2
tabulate : None
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.22.0
tzdata : 2024.1
qtpy : 2.4.1
pyqt5 : None

[rhshadrach]

Thanks for the report!

@lithomas1 - do you know if DataFrame.apply with numba was intended for non-reductions?