PyAthena >= 2.9.0 error in s3fs

[dblado]

Hi,

Using the following code works fine in pyathena[pandas]==2.8.0 but fails >= 2.9.0 with an error I haven't been able to identify the cause of. I used a fresh virtual environment and only installed the pyathena package.

I've compared this code to the examples and can't seem to find where I'm going wrong.

from pyathena import connect
from pyathena.pandas.cursor import PandasCursor

S3_STAGING_DIR = 'redacted'
AWS_ACCESS_KEY = "redacted"
AWS_SECRET_KEY = "redacted"
AWS_REGION = "redacted"
WORK_GROUP = "redacted"
ROLE_ARN = "redacted"

cursor = connect(aws_access_key_id=AWS_ACCESS_KEY, aws_secret_access_key=AWS_SECRET_KEY,
                 s3_staging_dir=S3_STAGING_DIR,
                 region_name=AWS_REGION,
                 work_group=WORK_GROUP,
                 role_arn=ROLE_ARN,
                 cursor_class=PandasCursor).cursor()

q=cursor.execute("select * from my_table limit 10")

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "python3.9/site-packages/pyathena/util.py", line 37, in _wrapper
    return wrapped(*args, **kwargs)
  File "python3.9/site-packages/pyathena/pandas/cursor.py", line 157, in execute
    self.result_set = AthenaPandasResultSet(
  File "python3.9/site-packages/pyathena/pandas/result_set.py", line 72, in __init__
    self._fs = self.__s3_file_system()
  File "python3.9/site-packages/pyathena/pandas/result_set.py", line 86, in __s3_file_system
    fs = fs.S3FileSystem(
  File "pyarrow/_s3fs.pyx", line 217, in pyarrow._s3fs.S3FileSystem.__init__
  File "stringsource", line 15, in string.from_py.__pyx_convert_string_from_py_std__in_string
TypeError: expected bytes, NoneType found

Environment: OSX running Python 3.9.7
$ pip freeze
aiobotocore==2.3.4
aiohttp==3.8.1
aioitertools==0.10.0
aiosignal==1.2.0
async-timeout==4.0.2
attrs==21.4.0
boto3==1.24.29
botocore==1.27.29
charset-normalizer==2.1.0
frozenlist==1.3.0
fsspec==2022.5.0
idna==3.3
jmespath==1.0.1
multidict==6.0.2
numpy==1.23.1
pandas==1.4.3
pyarrow==8.0.0
pyathena==2.9.6
python-dateutil==2.8.2
pytz==2022.1
s3fs==2022.5.0
s3transfer==0.6.0
six==1.16.0
tenacity==8.0.1
typing_extensions==4.3.0
urllib3==1.26.10
wrapt==1.14.1
yarl==1.7.2

[laughingman7743]

It seems that the role_arn cannot be specified. I will look into it.

[laughingman7743]

It may be related to the following issues.
https://issues.apache.org/jira/browse/ARROW-16864
apache/arrow#13455

[laughingman7743]

Since ExternalId specification was not supported in the first place, the following branch has been changed to support it.
#348