Python ‘pip’ and Windows registry corruption

Background

In my recent post about setting up a 64 bit Windows Django development environment, I described an initial problem with Virtualenv which then led me through a whole series of problems and fixes and finally to what I thought was a working 64 bit Python development environment. However, I recently realised that the original underlying issue still remained, such that any attempt to install a new package using pip would result in failure due to that same error message.

For example, when I tried to install Beautiful Soup via pip, I got the following error:

C:\> pip install beautifulsoup4
Downloading/unpacking beautifulsoup4
Cleaning up...
Exception:
Traceback (most recent call last):
File "C:\Python27\lib\site-packages\pip\basecommand.py", line 122, in main
status = self.run(options, args)
File "C:\Python27\lib\site-packages\pip\commands\install.py", line 278, in run

requirement_set.prepare_files(finder, force_root_egg_info=self.bundle, bundl
e=self.bundle)
File "C:\Python27\lib\site-packages\pip\req.py", line 1229, in prepare_files
req_to_install.run_egg_info()
File "C:\Python27\lib\site-packages\pip\req.py", line 292, in run_egg_info
logger.notify('Running setup.py (path:%s) egg_info for package %s' % (self.s
etup_py, self.name))
File "C:\Python27\lib\site-packages\pip\req.py", line 265, in setup_py
import setuptools
File "C:\Python27\lib\site-packages\setuptools\__init__.py", line 12, in
from setuptools.extension import Extension
File "C:\Python27\lib\site-packages\setuptools\extension.py", line 7, in
from setuptools.dist import _get_unpatched
File "C:\Python27\lib\site-packages\setuptools\dist.py", line 15, in
from setuptools.compat import numeric_types, basestring
File "C:\Python27\lib\site-packages\setuptools\compat.py", line 19, in
from SimpleHTTPServer import SimpleHTTPRequestHandler
File "C:\Python27\lib\SimpleHTTPServer.py", line 27, in
class SimpleHTTPRequestHandler(BaseHTTPServer.BaseHTTPRequestHandler):
File "C:\Python27\lib\SimpleHTTPServer.py", line 208, in SimpleHTTPRequestHand
ler
mimetypes.init() # try to read system mime.types
File "C:\Python27\lib\mimetypes.py", line 361, in init
db.read_windows_registry()
File "C:\Python27\lib\mimetypes.py", line 263, in read_windows_registry
with _winreg.OpenKey(hkcr, subkeyname) as subkey:
TypeError: must be string without null bytes or None, not str

Registry Problems

My initial research led me to believe the problem was due to missing registry entries that were required by ‘setuptools’ or ‘pip’ on 64 bit Windows systems. However, my attempts to follow the recommended registry fixes did not help at all. Neither did installing native 64 bit builds of ‘setuptools’ and ‘pip’ – even after successfully installing the 64 bit versions, they still failed to install any packages with exactly the same error.

I finally found a helpful discussion thread that led me to investigate the ‘mimetypes.py’ standard Python module, which is the origin of the traceback I keep getting. There was a suggestion that perhaps a corrupted registry key in ‘HKEY_CLASSES_ROOT’ could be breaking the ‘read_windows_registry()’ function.

I wrote the following script (for Python v2.7.6) to identify any problematic registry keys:

import _winreg as winreg

hkcr = winreg.OpenKey(winreg.HKEY_CLASSES_ROOT, '')

for i in range(999999):
    try:
        key = winreg.EnumKey(hkcr, i)
        if '\0' in key:
            print "Found null character in key: HKEY_CLASSES_ROOT/%s" % str(key).replace('\0', '*')
    except WindowsError:
        print "Finished!"
        break

This identified around 10 problematic keys which contained trailing null characters.

Alternative Solutions

Remove Corrupted Keys

The Windows Sysinternals RegDelNull tool can be used to easily find and delete any registry keys containing null characters. Although the problematic keys are being displayed in ‘HKEY_CLASSES_ROOT’, in fact they are being pulled in from ‘HKEY_LOCAL_MACHINE’ so that is where we must search:

F:\> RegDelNull.exe hklm -s

RegDelNull v1.10 - Delete Registry keys with embedded Nulls
Copyright (C) 2005-2006 Mark Russinovich
Sysinternals - www.sysinternals.com

Null-embedded key (Nulls are replaced by '*'):

HKLM\SOFTWARE\Classes\{03223D4D-1B28-4325-9A96-9C5A4C8EA8BC}*

Delete? (y/n)

If you are happy to completely remove any problematic keys, and accept any associated breakages that may occur, then this is the easiest solution.

Manually Patch ‘mimetypes.py’

I applied a manual 3 line hack-patch to the Python v2.7.6 ‘mimetypes.py’ module (around line 260) as follows:

with _winreg.OpenKey(_winreg.HKEY_CLASSES_ROOT, '') as hkcr:
    for subkeyname in enum_types(hkcr):
        try:
            if '\0' in subkeyname: # new
                print "Skipping bad key: %s" % subkeyname # new
                continue # new
            with _winreg.OpenKey(hkcr, subkeyname) as subkey:

With these 3 extra lines in place, I am now able to install new packages using ‘pip’ without any errors at all. This is potentially a good workaround for those who don’t want to edit their registry, although it is very nasty to hand-edit your core Python installation in this way. For the record, this is still an issue with the latest build of Python v3.4. Although this is caused by problematic registry entries, it would be nice if Python handled the situation more gracefully.

Update – 14th June 2015

This issue has since been reported in a couple of official Python bugs:

It looks like the issue has now been patched in both the Python 2.7 and 3.4 branches, although the status of Issue 22028 remains “open” so I don’t believe the fix is generally available quite yet.

31 thoughts on “Python ‘pip’ and Windows registry corruption

  1. Thank you very much for the info. I’m curious as to why this was so hard to find, yet the problem is very game-breaking. Is there anywhere else that talks about this same issue?
    Cheers.

    1. Hi Tom,

      Thanks for the feedback – I’m glad this was helpful. I’ve no idea why this issue is not better documented. I couldn’t find any decent information or any clear recommended solutions, so I gathered together all the separate nuggets of information I could find and came up with my own workaround (albeit not a very elegant one!). I’m getting quite a few hits on this post so I’m guessing others are also struggling to find proper solutions elsewhere.

      Cheers, Ian

  2. Thanks for the help, been looking for ages for a solution. Curiously only changing the mimetypes.py module worked, and popped up with 12 keys it skipped, yet when i ran the regdelnull hklm -s, it couldn’t find any.

    Cheers ๐Ÿ˜€

    1. Hi Pete,

      Thanks very much for the feedback – I’m really glad this article was helpful! I’m not too sure why ‘regdelnull’ didn’t track those problematic keys down, but at least the ‘mimetypes.py’ hack got things working for you.

      Cheers, Ian

  3. Having the same exact issue. Changing mimetypes.py makes it work. I just tried for a few hours to remove the bad registry key without success. RegDelNull.exe does not find anything, but your python tool finds a bad key. (This is on Windows 8) . If anyone has an idea how to remove the bad key from my registry let me know. @Ian: If you can, submit a patch! This cost me a few hours today. Thanks again for posting this.

    1. Hi Johannes,

      Thanks very much for the feedback – glad to help another satisfied customer! I’m not sure how to remove the bad key if ‘RegDelNull.exe’ won’t do it for you. I’ve never submitted a patch to Python itself before, but I’ll see if I can get anywhere with it…

      Thanks again, Ian

  4. Sure enough. Searched the net for a week and I finally found your blog! :] Thanks man. I had tried everything except hacking mimetypes.py myself and your workaround is the only thing that got me working so, to me it’s quite elegant! Keep up the good work and thanks again!

  5. Hi Ian!

    This saved the day for me… thanks!

    Just wanted to point out a typo in the code you quote for ‘mimetypes.pyโ€™ – the last line – ‘with _winreg.OpenKey(hkcr, subkeyname) as subkey:’ – needs to be indented 4 more spaces, to be consistent with the original code in the file – i.e. the 2nd ‘with’ needs to line up with the ‘if’

    Thanks again! Clare

  6. Thanks very much for this post, it saved me a lot of headache. I actually found it because I was searching for the offending key “03223D4D-1B28-4325-9A96-9C5A4C8EA8BC”, the exact same as in your RegDelNull output. Do you happen to have any idea which application could be the origin of those keys?

    1. Hi Tobias,

      Thanks very much for the feedback – I’m glad this post was helpful! Sorry but I have no idea which application is responsible for that particular key. That’s why I felt it was safer to ignore the null keys rather than delete them, because I don’t know which applications may break if I simply remove them all. This issue is a real pain!

      Ian

    2. Tobias – I have since discovered that some of my invalid registry keys may have been created by a software package called “FastPictureViewer Codec Pack”. Do you have that also? Ian

  7. This blog entry was a lifesaver!

    I hit this problem when trying to run the AWS S3 CLI to backup files from my Windows PC to Amazon S3.
    Running regdelnull.exe cleared Nulls from about 5 registry keys and solved the problem

    Like Ian Thomas I also have FastPictureViewer installed – so this may be causing the problem – but I did not record and trace the deleted keys so can’t help clarify further.

  8. Thanks – that was tremendously helpful! Used your script to find the keys, then RegDelNull to remove them.

  9. For your information: this bug is known to the Python developers and a patch is available. See: https://bugs.python.org/issue22028
    The patch has been applied to the main branch in March of 2015.
    See: https://github.com/python/cpython/commit/090e75f1c736967bedc75ae881fb15890ea36533
    At the time of writing this comment an official Python release containing this fix is not yet available for end users to download. Hopefully this will change soon.

Leave a Reply