我下了 Python 3.1.1 的 Windows 安装包：http://www.
,装好以后用它来运行一个测试脚本，出了如下的问题：

Traceback (most recent call last):
  File "D:\resource\programming\Python\MyPythonScripts\test.py", line 132, in vimcomplete
    cmpl.evalsource(filestr, lineidx)    # 改了
  File "D:\resource\programming\Python\MyPythonScripts\test.py", line 170, in evalsource
    sc = self.parser.parse(text, line)
  File "D:\resource\programming\Python\MyPythonScripts\test.py", line 567, in parse
    self.gen = tokenize.tokenize(buf.readline)
  File "D:\Program Files\Python31\lib\tokenize.py", line 369, in tokenize
    encoding, consumed = detect_encoding(readline)
  File "D:\Program Files\Python31\lib\tokenize.py", line 329, in detect_encoding
    if first.startswith(BOM_UTF8):
TypeError: Can't convert 'bytes' object to str implicitly

看起来问题出在标准库 tokenize.py 内部，329行那里。startswith() 函数的参数类型不对。查阅同时安装的帮助文档，里面确实写了，BOM_UTF8 是 bytes 类型的常数，可是 startswith() 函数要求参数是 str 类型。难道是标准库自己带错？

原因找到了，标准库没错（汗 ...），是用得不对。

正确用法：
--------------------------------------------------------------------------------
import tokenize

testbuffer= open('test.py', mode='rb')    # !! 一定要有 'b'
tokengenerator= tokenize.tokenize(testbuffer.readline)
--------------------------------------------------------------------------------
这是文档里的解释：

tokenize.tokenize(readline)
    The tokenize() generator requires one argument, readline, which must be a callable object which provides the same interface as the readline() method of built-in file objects (see section File Objects). Each call to the function should return one line of input as bytes.