Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

esptool stopped working with v4.2 (ESPTOOL-1002) #1059

Closed
1 task done
marcelstoer opened this issue Jan 28, 2025 · 9 comments
Closed
1 task done

esptool stopped working with v4.2 (ESPTOOL-1002) #1059

marcelstoer opened this issue Jan 28, 2025 · 9 comments

Comments

@marcelstoer
Copy link
Contributor

Operating System

macOS 14.7

Esptool Version

4.2

Python Version

3.12

Chip Description

ESP32-WROVER-B

Device Description

I am the author and main contributor of the popular esptool GUI "NodeMCU PyFlasher" (close to 500k downloads on GH): https://github.com/marcelstoer/nodemcu-pyflasher. Also, I initially created #157 in 2016 which Angus later rephrased in #208.

Side note, my tool just collects input from the GUI to build the esptool CLI flags and invokes it directly.

While analyzing the esptool v3 --> v4 upgrade issue in my project (marcelstoer/nodemcu-pyflasher#105), I noticed that esptool stopped working on v4.2 and everything more recent. When it fails, it simply hangs. I went through the v4.2 release notes, but nothing mentioned there would explain the behavior I am seeing.

I iterated through a number of esptool versions using the same hardware and the same CLI switches. Hence, the command created by my app was always the same: esptool.py --baud 921600 --after no_reset write_flash --flash_size detect --flash_mode dio 0x00000 /Users/marcelstoer/Data/my-app.bin. Please find the output below.

v3.3.3
The baseline. Used by the latest version of my app (released in 2021).

esptool.py v3.3.3
Found 7 serial ports
Serial port /dev/cu.wchusbserial54790238451
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting.....
Detecting chip type... ESP32
Chip is ESP32-D0WD (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 8c:4b:14:04:53:a4
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00000000 to 0x00128fff...
Compressed 1213984 bytes to 741190...
...

(I skipped the "Writing at" progress output and everything that follows)

v4.0.1
Despite the refactoring that came with v4, this version still works just like before. I also tested v4.1 which behaves the same as v4.0.1.

esptool.py v4.0.1
Found 7 serial ports
Serial port /dev/cu.wchusbserial54790238451
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting......
Detecting chip type... ESP32
Chip is ESP32-D0WD (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 8c:4b:14:04:53:a4
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 921600
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00000000 to 0x00128fff...
Compressed 1213984 bytes to 741190...
...

(I skipped the "Writing at" progress output and everything that follows)

v4.2
After dumping the MAC address, esptool just hangs i.e. there's no further output after MAC: 8c:4b:14:04:53:a4. The behavior is the same with v4.2.1 and v4.5.1.

esptool.py v4.2
Found 7 serial ports
Serial port /dev/cu.wchusbserial54790238451
Connecting....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting....
Detecting chip type... ESP32
Chip is ESP32-D0WD (revision 1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 8c:4b:14:04:53:a4

v4.8.1
The behavior remains the same as with v4.2+ expect for new error/warning printed after the MAC address. It says

If issues persist....

but it doesn't say which issue.

esptool.py v4.8.1
Found 6 serial ports
Serial port /dev/cu.wchusbserial54790238451
Connecting.....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting.....
Detecting chip type... ESP32
Chip is ESP32-D0WD (revision v1.0)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 8c:4b:14:04:53:a4

Note: If issues persist, try installing the WCH USB-to-Serial MacOS driver.

Hardware Configuration

ESP32-WROVER-B is embedded into the ThingPulse ESP32 ePulse Feather dev board. No components attached to it.

How is Esptool Run

from within PyFlasher

Full Esptool Command Line that Was Run

esptool.py --baud 921600 --after no_reset write_flash --flash_size detect --flash_mode dio 0x00000 /Users/marcelstoer/Data/my-app.bin

Esptool Output

see above

More Information

No response

Other Steps to Reproduce

No response

I Have Read the Troubleshooting Guide

  • I confirm I have read the troubleshooting guide.
@github-actions github-actions bot changed the title esptool stopped working with v4.2 esptool stopped working with v4.2 (ESPTOOL-1002) Jan 28, 2025
@radimkarnis
Copy link
Collaborator

radimkarnis commented Jan 28, 2025

Hi Marcel,
It's good to see you working on PyFlasher again!

First of all, I am finally working on addressing all of the ideas mentioned in #208. In the next major release v5, it should be much easier to work with esptool as a module and integrate it into a custom GUI. It's planned to be (hopefully) released in April/May (but the changes will start appearing on master soon). Feel free to offer more ideas about what would make your life easier, as I am working on that at the moment.

Now, to your issue - I am pretty sure this is caused by the way the stub flasher code is stored in esptool.
Up to version v4.1, the stub codes were all stored as base64 encoded text in the stub_flasher.py file. From v4.2, the stub flasher code is stored as individual JSON files in esptool/targets/stub_flasher/stub_flasher_XX.json.

After the MAC address is printed (MAC: 8c:4b:14:04:53:a4), the stub flasher should start to be uploaded - that's where it hangs. I guess it's looking for the JSON file but cannot find it - are you including the JSON files in PyFlasher?

If that turns out not to be the culprit, maybe we could look at the specific commit that causes this - git bisect can help to find the issue.

Let me know if this helps!

@marcelstoer
Copy link
Contributor Author

Thank you Radim for looking into this so quickly. It sounds as if had heard of the PyFlasher before?

From v4.2, the stub flasher code is stored as individual JSON files
...
are you including the JSON files in PyFlasher?

Hhhmm, I'm currently clutching at straws, but your description matches the behavior I'm seeing, thanks! PyFlasher is built with PyInstaller. I might have to tweak its configuration to include those "extra" files in the bundle (I vaguely remember there being such options).

@radimkarnis
Copy link
Collaborator

It sounds as if had heard of the PyFlasher before?

Of course, I have used it myself many times. Also, as it's one of the most popular GUIs for esptool, I have an eye on the project!

I'm currently clutching at straws, but your description matches the behavior I'm seeing

To verify, you can add the --no-stub argument and see if esptool still hangs or not.

PyFlasher is built with PyInstaller. I might have to tweak its configuration to include those "extra" files

Check the --add-data argument of PyInstaller, we also use it and have to include the stub files: https://github.com/espressif/esptool/blob/master/.github/workflows/build_esptool.yml#L47-L50

@DeeEmm
Copy link

DeeEmm commented Jan 29, 2025

I can add that I too see the same behaviour with a fork of PyFlasher that I am working on...

Image

It also hangs at the mac address.

However I also note that this only happens on the compiled binary. If I run PyFlasher from within VSCode, it works perfectly. This may support what @radimkarnis has reported

I will try the --add-data argument suggested and report back

@radimkarnis
Copy link
Collaborator

I don't know how PyFLasher handles the output, but I don't think esptool is "hanging". There is probably a FileNotFoundError error happening underneath, but it's maybe getting swallowed.

To support this, let's have a look at the new clue happening in v4.8.1 - Note: If issues persist, try installing the WCH USB-to-Serial MacOS driver..

This note is unrelated to the issue, but happens in the except block of try: esp.run_stub(). That means run_stub() is definitely throwing an exception (probably FileNotFoundError), raise is definitely happening as well, but PyFlasher swallows it.

                try:
                    esp = esp.run_stub()
                except Exception:
                    # The CH9102 bridge (PID: 0x55D4) can have issues on MacOS
                    if sys.platform == "darwin" and esp._get_pid() == 0x55D4:
                        print()
                        print(
                            "If issues persist, "
                            "try installing the WCH USB-to-Serial MacOS driver."
                        )
                   raise

@DeeEmm
Copy link

DeeEmm commented Jan 29, 2025

Yes hanging is perhaps a poor choice of words. Execution appeared to stall. Unfortunately I was unable to see what was actually going as logs did not reveal anything to me. But some research highlighted that others have had the same issue and it was as you described related to the inclusion of the stubs files.

So after a time and some experimentation I managed to get my fork of pyflasher working with esptool v4.8.1

Command: esptool.py --chip esp32 --port /dev/cu.usbserial-14410 --baud 460800 --after hard_reset write_flash --flash_size detect --flash_mode dio 0x0000 /Users/mick/Documents/ESP32 Projects/DIY-Flow-Bench/ESP32/DIY-Flow-Bench/release/V2.0-RC8_2501280085_install.bin

esptool.py v4.8.1
Serial port /dev/cu.usbserial-14410
Connecting.....
Detecting chip type... Unsupported detection protocol, switching and trying again...
Connecting.....
Detecting chip type... ESP32
Chip is ESP32-D0WD-V3 (revision v3.1)
Features: WiFi, BT, Dual Core, 240MHz, VRef calibration in efuse, Coding Scheme None
Crystal is 40MHz
MAC: 1c:69:20:ce:88:e4
Uploading stub...
Running stub...
Stub running...
Changing baud rate to 460800
Changed.
Configuring flash size...
Auto-detected Flash size: 4MB
Flash will be erased from 0x00000000 to 0x003fffff...
Compressed 4194304 bytes to 809686...
Wrote 4194304 bytes (809686 compressed) at 0x00000000 in 27.7 seconds (effective 1213.2 kbit/s)...
Hash of data verified.

Leaving...
Hard resetting via RTS pin...

This is my requirements.txt

esptool>=4.0    
pyserial>=3.5
wxPython>=4.2.2
PyInstaller>=6.11.1
httplib2>=0.22.0
pyinstaller-versionfile>=2.0.0

The main changes were to add the stubs folders to the 'datas' section in the specs file... (pyflasher uses specs files to handle CLI options)

'build-on-mac.spec'

# -*- mode: python -*-

block_cipher = None

added_files = [
   ( "/Users/mick/Documents/ESP32 Projects/DIYFB-firmware-flasher/.venv/lib/python3.13/site-packages/esptool/targets/stub_flasher/1", "./esptool/targets/stub_flasher/1"),
   ( "/Users/mick/Documents/ESP32 Projects/DIYFB-firmware-flasher/.venv/lib/python3.13/site-packages/esptool/targets/stub_flasher/2", "./esptool/targets/stub_flasher/2")
   ]

imported_files = [
   ("wxpython.py"),
   ("pyserial")
]


a = Analysis(['DIYFB-Firmware-Flasher.py'],
             binaries=None,
             datas=added_files,
             hiddenimports=imported_files,
             hookspath=[],
             runtime_hooks=[],
             excludes=[],
             win_no_prefer_redirects=False,
             win_private_assemblies=False,
             cipher=block_cipher)
pyz = PYZ(a.pure, a.zipped_data,
             cipher=block_cipher)
exe = EXE(pyz,
          a.scripts,
          a.binaries,
          a.zipfiles,
          a.datas,
          name='DIYFB-Firmware-Flasher',
          debug=True,
          strip=False,
          upx=True,
          console=False , icon='images/icon-256.icns')
app = BUNDLE(exe,
             name='DIYFB-Firmware-Flasher.app',
             version='1.0.1',
             icon='./images/icon-256.icns',
             bundle_identifier='com.DeeEmm.DIYFB-Firmware-Flasher')

Filepaths could obviously be handled a bit more elegantly but it is working 😃

@marcelstoer
Copy link
Contributor Author

@radimkarnis

I don't know how PyFLasher handles the output, but I don't think esptool is "hanging". There is probably a FileNotFoundError error happening underneath, but it's maybe getting swallowed.

There's a minimally clever piece of code that redirects standard out to a UI component (called RedirectText). Hence, any esptool output that makes it onto stdout should be visible to PyFlasher users.

PyFlasher does not handle esptool exceptions explicitly except SerialException and doesn't configure logging. From what I understand, PyInstaller's default behavior is then, that those other exceptions go to standard error. They are lost unless you run the executable from the terminal.
Maybe I should use sys.excepthook and display such exceptions in a UI overlay or the like. From what I know, PyInstaller does support Python's global exception handler.

@DeeEmm thanks for testing Radim's proposal!

@radimkarnis
Copy link
Collaborator

@marcelstoer cool! I believe we can close this ticket, correct?

@marcelstoer
Copy link
Contributor Author

Thanks again for all your support! The fix landed with marcelstoer/nodemcu-pyflasher@2d5daab.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

3 participants