Skip to content

[encodings] Accept buffer inputs in Codec.decode for charmap encodings - #16686

Open
huuan26 wants to merge 2 commits into
python:mainfrom
huuan26:fix/encodings-cp125x-decode-buffer
Open

huuan26 wants to merge 2 commits into
python:mainfrom
huuan26:fix/encodings-cp125x-decode-buffer

Conversation

@huuan26

@huuan26 huuan26 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

All charmap encodings generated via Tools/unicode/gencodec.py implement Codec.decode by delegating directly to codecs.charmap_decode(input, errors, decoding_table) (which accepts any ReadableBuffer, just like IncrementalDecoder.decode in the same modules and cp1252.pyi).

This PR updates Codec.decode across all remaining gencodec.py-generated charmap encoding stubs in stdlib/encodings/ (cp*, hp_roman8, iso8859_*, koi8_*, kz1048, mac_*, palmos, ptcp154, tis_620) to accept ReadableBuffer instead of bytes.

@github-actions

This comment has been minimized.

@srittau

srittau commented Nov 4, 2026

Copy link
Copy Markdown
Collaborator

Do you have a link to the implementation in CPython to verify this?

@huuan26

huuan26 commented Nov 5, 2026 •

Copy link
Copy Markdown
Contributor Author

Here are the links to the CPython implementation:

  1. Lib/encodings/cp1250.py – cp1258.py (generated via Tools/unicode/gencodec.py):
    Codec.decode delegates directly to codecs.charmap_decode(input, errors, decoding_table) (just like IncrementalDecoder.decode, which is already typed with ReadableBuffer in these stubs):

  2. Modules/_codecsmodule.c (_codecs.charmap_decode):
    In Argument Clinic, charmap_decode takes data: Py_buffer, accepting any ReadableBuffer (bytes, bytearray, memoryview, etc.), which matches _codecs.charmap_decode(data: ReadableBuffer, ...) in stdlib/_codecs.pyi:

@JelleZijlstra

Copy link
Copy Markdown
Member

Should we do this for a lot more of the encodings modules?

@huuan26 huuan26 changed the title [encodings] Accept buffer inputs in Codec.decode for cp1250-cp1258 [encodings] Accept buffer inputs in Codec.decode for charmap encodings Nov 5, 2026
@huuan26

huuan26 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Good call @JelleZijlstra! I audited all stdlib/encodings/*.pyi stubs and found 61 other gencodec.py-generated charmap encoding modules (cp037–cp1140, hp_roman8, iso8859_*, koi8_*, kz1048, mac_*, palmos, ptcp154, tis_620) that also delegate Codec.decode to codecs.charmap_decode(input, errors, decoding_table) (and already imported ReadableBuffer for IncrementalDecoder.decode).

I have updated all 61 remaining charmap encoding stubs in bc662da so they are all consistent.

@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

According to mypy_primer, this change has no effect on the checked open source code. 🤖🎉

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants