Getting broken pipe error on github runner, but not on unmodified self hosted aws ec2

[jack-rep]

Why are you starting this discussion?

Bug

What GitHub Actions topic or product is this about?

Actions Runner

Discussion Details

When running a job on ubuntu-latest (or 24 or 22) we get an error on every step (but continues to run until it hangs on cargo build step). While running same job finishes without any errors on self-hosted unmodified runner using aws t2.medium ec2.

The job runs molecule which creates another server and runs tests on that other server.

<3.149.244.102> ESTABLISH SSH CONNECTION FOR USER: ubuntu
<3.149.244.102> SSH: EXEC ssh -vvv -C -o ControlMaster=auto -o ControlPersist=60s -o StrictHostKeyChecking=no -o Port=22 -o 'IdentityFile="/home/runner/.ansible/tmp/molecule.<REDACTED>"' -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="ubuntu"' -o ConnectTimeout=10 -o ControlPersist=30m -o ControlMaster=yes -o ConnectTimeout=99999 -o 'ControlPath="/home/runner/.ansible/cp/%h-%p-%r"' 3.149.244.102 '/bin/sh -c '"'"'( umask 77 && mkdir -p "` echo /home/ubuntu/.ansible/tmp `"&& mkdir "` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1751913589.1488814-2971-21672251039940 `" && echo ansible-tmp-1751913589.1488814-2971-21672251039940="` echo /home/ubuntu/.ansible/tmp/ansible-tmp-1751913589.1488814-2971-21672251039940 `" ) && sleep 0'"'"''
<3.149.244.102> (0, b'ansible-tmp-1751913589.1488814-2971-21672251039940=/home/ubuntu/.ansible/tmp/ansible-tmp-1751913589.1488814-2971-21672251039940\n', b"OpenSSH_9.6p1 Ubuntu-3ubuntu13.12, OpenSSL 3.0.13 30 Jan 2024\r\ndebug1: Reading configuration data /etc/ssh/ssh_config\r\ndebug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files\r\ndebug1: /etc/ssh/ssh_config line 21: Applying options for *\r\ndebug2: resolve_canonicalize: hostname 3.149.244.102 is address\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts' -> '/home/runner/.ssh/known_hosts'\r\ndebug3: expanded UserKnownHostsFile '~/.ssh/known_hosts2' -> '/home/runner/.ssh/known_hosts2'\r\ndebug1: auto-mux: Trying existing master at '/home/runner/.ansible/cp/3.149.244.102-22-ubuntu'\r\ndebug2: fd 3 setting O_NONBLOCK\r\ndebug2: mux_client_hello_exchange: master version 4\r\ndebug3: mux_client_forwards: request forwardings: 0 local, 0 remote\r\ndebug3: mux_client_request_session: entering\r\ndebug3: mux_client_request_alive: entering\r\ndebug3: mux_client_request_alive: done pid = 2409\r\ndebug3: mux_client_request_session: session request sent\r\ndebug1: mux_client_request_session: master session id: 2\r\ndebug3: mux_client_read_packet_timeout: read header failed: Broken pipe\r\ndebug2: Received exit status from master 0\r\n")

Tried extending ConnectTimeout as you can see and ControlPersist, but that didn't help. Any idea why github runner errors and hangs on cargo build like this, but normal t2.medium aws ec2 works?

[github-actions[bot]]

💬 Your Product Feedback Has Been Submitted 🎉

Thank you for taking the time to share your insights with us! Your feedback is invaluable as we build a better GitHub experience for all our users.

Here's what you can expect moving forward ⏩

• Your input will be carefully reviewed and cataloged by members of our product teams. 
  • Due to the high volume of submissions, we may not always be able to provide individual responses.
  • Rest assured, your feedback will help chart our course for product improvements.
• Other users may engage with your post, sharing their own perspectives or experiences. 
• GitHub staff may reach out for further clarification or insight. 
  • We may 'Answer' your discussion if there is a current solution, workaround, or roadmap/changelog post related to the feedback.

Where to look to see what's shipping 👀

• Read the Changelog for real-time updates on the latest GitHub features, enhancements, and calls for feedback.
• Explore our Product Roadmap, which details upcoming major releases and initiatives.

What you can do in the meantime 💻

• Upvote and comment on other user feedback Discussions that resonate with you.
• Add more information at any point! Useful details include: use cases, relevant labels, desired outcomes, and any accompanying screenshots.

As a member of the GitHub community, your participation is essential. While we can't promise that every suggestion will be implemented, we want to emphasize that your feedback is instrumental in guiding our decisions and priorities.

Thank you once again for your contribution to making GitHub even better! We're grateful for your ongoing support and collaboration in shaping the future of our platform. ⭐

[Tech-Nomadic-X]

Disable SSH multiplexing in your Ansible/Molecule configuration:
yaml

ansible_ssh_args: -o ControlMaster=no -o ControlPersist=no

Increase timeouts more aggressively:
yaml

ansible_ssh_args: -o ConnectTimeout=60 -o ServerAliveInterval=60 -o ServerAliveCountMax=10

Try using the paramiko connection plugin instead of OpenSSH:
yaml

ansible_connection: paramiko

Split the cargo build into a separate step or use a build cache to reduce the time needed

Check for network restrictions - GitHub-hosted runners might have different outbound connection policies

Additional Debugging Steps:

Add -vvv to your Molecule command to get more verbose output

Try running a simple long-running command (like sleep 300) via Ansible to see if it's specific to cargo or any long operation

Check if the issue persists with Ubuntu 20.04 runners (ubuntu-20.04)

[jack-rep]

thanks! Will try to narrow down which option exactly helps, but it worked.

[jack-rep]

So adding the following to the provisioner section of molecule.yml helped solve the issue. ConnectTimeout helped fix the broken pipe error, however ServerAliveInterval helped fix the hanging of the cargo build step in molecule. Thanks again!

connection_options:
    ansible_ssh_common_args: -o ServerAliveInterval=60 -o ConnectTimeout=60