AttributeError: module 'select' has no attribute 'epoll'

[kushalmraut]

version : trio==0.22.2

[2023-10-10 12:06:56 +0000] [4254] [INFO] Worker exiting (pid: 4254)
[2023-10-10 12:07:00 +0000] [4232] [INFO] Shutting down: Master
[2023-10-10 12:07:00 +0000] [4232] [INFO] Reason: Worker failed to boot.
[2023-10-10 12:07:04 +0000] [4264] [DEBUG] Current configuration:
 config: None
 bind: ['unix:/run/anahit.sock']
 backlog: 2048
 workers: 1
 worker_class: gevent
 threads: 1
 worker_connections: 1000
 max_requests: 0
 max_requests_jitter: 0
 timeout: 120
 graceful_timeout: 30
 keepalive: 2
 limit_request_line: 4094
 limit_request_fields: 100
 limit_request_field_size: 8190
 reload: False
 reload_engine: auto
 reload_extra_files: []
 spew: False
 check_config: False
 preload_app: False
 sendfile: None
 reuse_port: False
 chdir: /home/anahit_web/anahit_website/app
 daemon: False
 raw_env: []
 pidfile: None
 worker_tmp_dir: None
 user: 1000
 group: 33
 umask: 0
 initgroups: False
 tmp_upload_dir: None
 secure_scheme_headers: {'X-FORWARDED-PROTOCOL': 'ssl', 'X-FORWARDED-PROTO': 'https', 'X-FORWARDED-SSL': 'on'}
 forwarded_allow_ips: ['127.0.0.1']
 accesslog: /home/anahit_web/logs/access.log
 disable_redirect_access_to_syslog: False
 access_log_format: %(h)s %(l)s %(u)s %(t)s "%(r)s" %(s)s %(b)s "%(f)s" "%(a)s"
 errorlog: /home/anahit_web/logs/anahit_gunicorn.log
 loglevel: debug
 capture_output: True
 logger_class: gunicorn.glogging.Logger
 logconfig: None
 logconfig_dict: {}
 syslog_addr: udp://localhost:514
 syslog: False
 syslog_prefix: None
 syslog_facility: user
 enable_stdio_inheritance: True
 statsd_host: None
 dogstatsd_tags: 
 statsd_prefix: 
 proc_name: None
 default_proc_name: anahit.wsgi:application
 pythonpath: None
 paste: None
 on_starting: <function OnStarting.on_starting at 0x7f93d1e15440>
 on_reload: <function OnReload.on_reload at 0x7f93d1e15580>
 when_ready: <function WhenReady.when_ready at 0x7f93d1e156c0>
 pre_fork: <function Prefork.pre_fork at 0x7f93d1e15800>
 post_fork: <function Postfork.post_fork at 0x7f93d1e15940>
 post_worker_init: <function PostWorkerInit.post_worker_init at 0x7f93d1e15a80>
 worker_int: <function WorkerInt.worker_int at 0x7f93d1e15bc0>
 worker_abort: <function WorkerAbort.worker_abort at 0x7f93d1e15d00>
 pre_exec: <function PreExec.pre_exec at 0x7f93d1e15e40>
 pre_request: <function PreRequest.pre_request at 0x7f93d1e15f80>
 post_request: <function PostRequest.post_request at 0x7f93d1e16020>
 child_exit: <function ChildExit.child_exit at 0x7f93d1e16160>
 worker_exit: <function WorkerExit.worker_exit at 0x7f93d1e162a0>
 nworkers_changed: <function NumWorkersChanged.nworkers_changed at 0x7f93d1e163e0>
 on_exit: <function OnExit.on_exit at 0x7f93d1e16520>
 proxy_protocol: False
 proxy_allow_ips: ['127.0.0.1']
 keyfile: None
 certfile: None
 ssl_version: 2
 cert_reqs: 0
 ca_certs: None
 suppress_ragged_eofs: True
 do_handshake_on_connect: False
 ciphers: None
 raw_paste_global_conf: []
 strip_header_spaces: False
[2023-10-10 12:07:04 +0000] [4264] [INFO] Starting gunicorn 20.0.2
[2023-10-10 12:07:04 +0000] [4264] [DEBUG] Arbiter booted
[2023-10-10 12:07:04 +0000] [4264] [INFO] Listening at: unix:/run/anahit.sock (4264)
[2023-10-10 12:07:04 +0000] [4264] [INFO] Using worker: gevent
[2023-10-10 12:07:04 +0000] [4285] [INFO] Booting worker with pid: 4285
[2023-10-10 12:07:04 +0000] [4264] [DEBUG] 1 workers
[2023-10-10 12:07:21 +0000] [4285] [ERROR] Exception in worker process
Traceback (most recent call last):
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/arbiter.py", line 583, in spawn_worker
  worker.init_process()
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/workers/ggevent.py", line 162, in init_process
  super().init_process()
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/workers/base.py", line 119, in init_process
  self.load_wsgi()
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/workers/base.py", line 144, in load_wsgi
  self.wsgi = self.app.wsgi()
        ^^^^^^^^^^^^^^^
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/app/base.py", line 66, in wsgi
  self.callable = self.load()
          ^^^^^^^^^^^
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 49, in load
  return self.load_wsgiapp()
      ^^^^^^^^^^^^^^^^^^^
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
  return util.import_app(self.app_uri)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/util.py", line 358, in import_app
  mod = importlib.import_module(module)
     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/usr/local/lib/python3.11/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "<frozen importlib._bootstrap>", line 1206, in _gcd_import
 File "<frozen importlib._bootstrap>", line 1178, in _find_and_load
 File "<frozen importlib._bootstrap>", line 1149, in _find_and_load_unlocked
 File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
 File "<frozen importlib._bootstrap_external>", line 940, in exec_module
 File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
 File "/home/anahit_web/anahit_website/app/anahit/wsgi.py", line 16, in <module>
  application = get_wsgi_application()
         ^^^^^^^^^^^^^^^^^^^^^^
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/django/core/wsgi.py", line 12, in get_wsgi_application
  django.setup(set_prefix=False)
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/django/__init__.py", line 24, in setup
  apps.populate(settings.INSTALLED_APPS)
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/django/apps/registry.py", line 124, in populate
  app_config.ready()
 File "/home/anahit_web/anahit_website/app/apps/users/apps.py", line 9, in ready
  from . import signals
 File "/home/anahit_web/anahit_website/app/apps/users/signals.py", line 11, in <module>
  from common.utils import send_email_notification
 File "/home/anahit_web/anahit_website/app/common/utils.py", line 28, in <module>
  from discord_webhook import DiscordWebhook
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/discord_webhook/__init__.py", line 5, in <module>
  from .async_webhook import AsyncDiscordWebhook
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/discord_webhook/async_webhook.py", line 13, in <module>
  import httpx # noqa
  ^^^^^^^^^^^^
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpx/__init__.py", line 2, in <module>
  from ._api import delete, get, head, options, patch, post, put, request, stream
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpx/_api.py", line 4, in <module>
  from ._client import Client
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpx/_client.py", line 30, in <module>
  from ._transports.default import AsyncHTTPTransport, HTTPTransport
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpx/_transports/default.py", line 30, in <module>
  import httpcore
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpcore/__init__.py", line 1, in <module>
  from ._api import request, stream
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpcore/_api.py", line 5, in <module>
  from ._sync.connection_pool import ConnectionPool
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpcore/_sync/__init__.py", line 1, in <module>
  from .connection import HTTPConnection
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpcore/_sync/connection.py", line 12, in <module>
  from .._synchronization import Lock
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/httpcore/_synchronization.py", line 13, in <module>
  import trio
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/trio/__init__.py", line 19, in <module>
  from ._core import TASK_STATUS_IGNORED as TASK_STATUS_IGNORED # isort: skip
  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/trio/_core/__init__.py", line 21, in <module>
  from ._local import RunVar
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/trio/_core/_local.py", line 5, in <module>
  from . import _run
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/trio/_core/_run.py", line 2543, in <module>
  from ._io_epoll import EpollIOManager as TheIOManager
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/trio/_core/_io_epoll.py", line 189, in <module>
  class EpollIOManager:
 File "/home/anahit_web/anahit_311/lib/python3.11/site-packages/trio/_core/_io_epoll.py", line 190, in EpollIOManager
  _epoll = attr.ib(factory=select.epoll)
               ^^^^^^^^^^^^
AttributeError: module 'select' has no attribute 'epoll'

I am getting above error if I specify the workers in gunicorn config.

ExecStart= gunicorn_path \
          --log-level debug \
          --enable-stdio-inheritance \
          --capture-output \
          --access-logfile access_log_path \
          --error-logfile error_log_path\
          --worker-class gevent \
          --workers 2 \
          --timeout 120 \
          --bind unix: <socket> \
          project.wsgi:application

gunicorn service file

[altendky]

Are you running Linux? It appears so and https://docs.python.org/3.11/library/select.html#select.epoll reads like it should be fairly available. Maybe the wrong select module is getting loaded. Could you add raise Exception(str(select)) right before the failing line? That will tell where the loaded select module is located.

[kushalmraut]

@altendky thank you for reply, Yes I am running linux based operating system.
I have added raise Exception(str(select)) right before the failing line Exception: <module 'select' from '/usr/local/lib/python3.11/lib-dynload/select.cpython-311-x86_64-linux-gnu.so'> is the exception I am getting.

[altendky]

So Trio does assume that Python will have select.epoll() in Linux.

https://github.com/python-trio/trio/blob/v0.22.2/trio/_core/_run.py#L2541-L2543

elif sys.platform == "linux" or (not TYPE_CHECKING and hasattr(select, "epoll")):
    from ._generated_io_epoll import *
    from ._io_epoll import EpollIOManager as TheIOManager

But Python checks to see if it is available on the system at compile time. It looks like this is set via the configure script.

https://github.com/python/cpython/blob/v3.11.6/Modules/selectmodule.c#L1230

#ifdef HAVE_EPOLL

Could you review this 'script' to make sure you are comfortable running it, and if so, share back the output? (after reviewing the output for anything you might be concerned about sharing :]) Consider the path to python and if that is wrong for the python you are using, correct it.

echo ---- python select && /usr/bin/python3.11 -c 'import select; print(select); print(getattr(select, "epoll", None))' && echo ---- python config && python -c 'import sysconfig; config = sysconfig.get_config_vars(); print({key: value for key, value in config.items() if "EPOLL" in key})' &&echo ---- uname && uname -a && echo ---- lsb_release && lsb_release -a && echo ---- boot config && grep -r EPOLL /boot/config* && echo ---- proc && grep -ri epoll /proc/sys/kernel/

For example, I get this output (for my Python 3.10 instead) showing that epoll seems to be enabled in both the kernel and in Python.

$ echo ---- python select && /usr/bin/python3.10 -c 'import select; print(select); print(getattr(select, "epoll", None))' && echo ---- python config && python -c 'import sysconfig; config = sysconfig.get_config_vars(); print({key: value for key, value in config.items() if "EPOLL" in key})' &&echo ---- uname && uname -a && echo ---- lsb_release && lsb_release -a && echo ---- boot config && grep -r EPOLL /boot/config* && echo ---- proc && grep -ri epoll /proc/sys/kernel/
---- python select
<module 'select' (built-in)>
<class 'select.epoll'>
---- python config
{'HAVE_EPOLL': 1, 'HAVE_EPOLL_CREATE1': 1, 'HAVE_SYS_EPOLL_H': 1}
---- uname
Linux p1 6.2.0-36-generic #37~22.04.1-Ubuntu SMP PREEMPT_DYNAMIC Mon Oct  9 15:34:04 UTC 2 x86_64 x86_64 x86_64 GNU/Linux
---- lsb_release
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
---- boot config
/boot/config-5.19.0-46-generic:CONFIG_EPOLL=y
/boot/config-6.2.0-35-generic:CONFIG_EPOLL=y
/boot/config-6.2.0-36-generic:CONFIG_EPOLL=y
---- proc
grep: /proc/sys/kernel/apparmor_display_secid_mode: Permission denied
grep: /proc/sys/kernel/apparmor_restrict_unprivileged_userns: Permission denied
grep: /proc/sys/kernel/apparmor_restrict_unprivileged_userns_complain: Permission denied
grep: /proc/sys/kernel/apparmor_restrict_unprivileged_userns_force: Permission denied
grep: /proc/sys/kernel/cad_pid: Permission denied
grep: /proc/sys/kernel/unprivileged_userns_apparmor_policy: Permission denied
grep: /proc/sys/kernel/usermodehelper/bset: Permission denied
grep: /proc/sys/kernel/usermodehelper/inheritable: Permission denied

[jakkdl]

grep -ri epoll /proc/sys/kernel/

I get this output (for my Python 3.10 instead) showing that epoll seems to be enabled in both the kernel and in Python.

grep: /proc/sys/kernel/apparmor_display_secid_mode: Permission denied
grep: /proc/sys/kernel/apparmor_restrict_unprivileged_userns: Permission denied
grep: /proc/sys/kernel/apparmor_restrict_unprivileged_userns_complain: Permission denied
grep: /proc/sys/kernel/apparmor_restrict_unprivileged_userns_force: Permission denied
grep: /proc/sys/kernel/cad_pid: Permission denied
grep: /proc/sys/kernel/unprivileged_userns_apparmor_policy: Permission denied
grep: /proc/sys/kernel/usermodehelper/bset: Permission denied
grep: /proc/sys/kernel/usermodehelper/inheritable: Permission denied

no, you're just getting a bunch of permission errors. Running it on my system with sudo I'm not getting any output, and the boot config similarly doesn't work on my system. (Arch Linux)

zcat /proc/config.gz | grep -i EPOLL

Was suggested by the internet for checking EPOLL availability though, and works on my system - so @kushalmraut might want to run that in addition and check the output in case their system differs from altkendy.

[kushalmraut]

@altendky

Output for the script is:

$ echo ---- python select && /home/anahit_web/venv_311/bin/python -c 'import select; print(select); print(getattr(select, "epoll", None))' && echo ---- python config && python -c 'import sysconfig; config = sysconfig.get_config_vars(); print({key: value for key, value in config.items() if "EPOLL" in key})' &&echo ---- uname && uname -a && echo ---- lsb_release && lsb_release -a && echo ---- boot config && grep -r EPOLL /boot/config* && echo ---- proc && grep -ri epoll /proc/sys/kernel/
---- python select
<module 'select' from '/usr/local/lib/python3.11/lib-dynload/select.cpython-311-x86_64-linux-gnu.so'>
<class 'select.epoll'>
---- python config
{'HAVE_EPOLL': 1, 'HAVE_EPOLL_CREATE1': 1, 'HAVE_SYS_EPOLL_H': 1}
---- uname
Linux ip-172-31-26-228 5.4.0-1103-aws #111~18.04.1-Ubuntu SMP Tue May 23 20:04:10 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
---- lsb_release
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic
---- boot config
/boot/config-5.4.0-1060-aws:CONFIG_EPOLL=y
/boot/config-5.4.0-1094-aws:CONFIG_EPOLL=y
/boot/config-5.4.0-1101-aws:CONFIG_EPOLL=y
/boot/config-5.4.0-1103-aws:CONFIG_EPOLL=y
---- proc
grep: /proc/sys/kernel/cad_pid: Permission denied
grep: /proc/sys/kernel/unprivileged_userns_apparmor_policy: Permission denied
grep: /proc/sys/kernel/usermodehelper/bset: Permission denied
grep: /proc/sys/kernel/usermodehelper/inheritable: Permission denied

[kushalmraut]

@jakkdl
I couldn't find /proc/config.gz a file.

[A5rocks]

I'm struggling to reconcile:

  _epoll = attr.ib(factory=select.epoll)
               ^^^^^^^^^^^^
AttributeError: module 'select' has no attribute 'epoll'

And yet getattr(select, "epoll", None) seems to work...

(I don't have anything productive to add, just found this puzzling).

[jakkdl]

yeah this is very strange. It feels like there might be a bug in gunicorn or something being misconfigured somehow such that the python/select.epoll being invoked by trio is not the same python as the one that could access it np from the command line. I think you'll have to do further debugging on your end, e.g. liberally adding print(getattr(select, "epoll")) to your code and/or the installed library code of gunicorn/trio.
It does not really sound like a problem on trios end though, you could try asking folks from gunicorn.

[kushalmraut]

Please let me elaborate in detail.
I started getting this error when I upgraded python version for my project from 3.9 to 3.11 and my django version 3.2 to 4.2.

This is my os

Linux #111~18.04.1-Ubuntu SMP x86_64 x86_64 x86_64 GNU/Linux

when i specify workers in gunicorn configurations i get this error.

--worker-class gevent \
--workers 2 \

I search for this issue and followed following links but unable to fix it.
I have also raised the issue on gunicorn github.

followed references.
https://www.mail-archive.com/debian-bugs-dist@lists.debian.org/msg1918516.html
urllib3/urllib3#1104

Please let me know if you find solution or root of cause.

[jakkdl]

I started getting this error when I upgraded python version for my project from 3.9 to 3.11 and my django version 3.2 to 4.2.

hm, is it a problem with virtualenvs or similar? Some questions to help you narrow down the problem: Do you also get it in CI? Tried a clean virtualenv? What distro? How did you do the python version upgrade? If you create a new project can you reproduce the problem? Does the problem disappear if you revert the upgrade?

[kushalmraut]

I started getting this error when I upgraded python version for my project from 3.9 to 3.11 and my django version 3.2 to 4.2.

hm, is it a problem with virtualenvs or similar? Some questions to help you narrow down the problem: Do you also get it in CI? Tried a clean virtualenv? What distro? How did you do the python version upgrade? If you create a new project can you reproduce the problem? Does the problem disappear if you revert the upgrade?

I don't have CL if you're referring to the continuous integration.

I have created a new virtualenv with python 3.11 while upgrading the packages instate of upgrading previous virtualenv.

distro

VERSION="18.04.6 LTS (Bionic Beaver)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 18.04.6 LTS"
VERSION_ID="18.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=bionic
UBUNTU_CODENAME=bionic

I have installed a 3.11 separately used this reference for python installation

The issue is reproduced when new project is created. I have two servers stage and main and issue is for both.

The problem is disappear when i switch to the old virtualenv.

[jakkdl]

I think you'll have to do further debugging on your end, e.g. liberally adding print(getattr(select, "epoll")) to your code and/or the installed library code of gunicorn/trio.

did you ever try this?
Clarifying the last part, open up files like /home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/arbiter.py in a text editor and modify them to add print statements and the like. The backtrace is pretty huge so you might have to modify a lot of places to figure out if/when the getattr call breaks.

[kushalmraut]

I think you'll have to do further debugging on your end, e.g. liberally adding print(getattr(select, "epoll")) to your code and/or the installed library code of gunicorn/trio.

did you ever try this? Clarifying the last part, open up files like /home/anahit_web/anahit_311/lib/python3.11/site-packages/gunicorn/arbiter.py in a text editor and modify them to add print statements and the like. The backtrace is pretty huge so you might have to modify a lot of places to figure out if/when the getattr call breaks.

Yes I have tried adding print(getattr(select, "epoll")) before _epoll = attr.ib(factory=select.epoll) it is giving the same error.

File "/home/anahit_web/venv_311/lib/python3.11/site-packages/trio/_core/_io_epoll.py", line 190, in EpollIOManager
  print('getattr(select, "epoll")*****************', getattr(select, "epoll"))
                            ^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'select' has no attribute 'epoll'

I have also deleted the python 3.11 and old vertualenv and installed python 3.11 again and created new vertualenv and tried it again but getting the same error.

I have printed dir(select) before the line causing the error. below is the output this might help.

['EPOLLERR', 'EPOLLET', 'EPOLLEXCLUSIVE', 'EPOLLHUP', 'EPOLLIN', 'EPOLLMSG', 'EPOLLONESHOT', 'EPOLLOUT', 'EPOLLPRI', 'EPOLLRDBAND', 'EPOLLRDHUP', 'EPOLLRDNORM', 'EPOLLWRBAND', 'EPOLLWRNORM', 'EPOLL_CLOEXEC', 'PIPE_BUF', 'POLLERR', 'POLLHUP', 'POLLIN', 'POLLMSG', 'POLLNVAL', 'POLLOUT', 'POLLPRI', 'POLLRDBAND', 'POLLRDHUP', 'POLLRDNORM', 'POLLWRBAND', 'POLLWRNORM', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'error', 'poll', 'select']

[jakkdl]

Yes I have tried adding print(getattr(select, "epoll")) before _epoll = attr.ib(factory=select.epoll) it is giving the same error.

and then add it to every other location in the backtrace, to figure out where it breaks. The big question mark is why you can run it in the python prompt w/o error but it breaking deep down in the call stack when trio tries to do it