Please note:The SCons wiki is in read-only mode due to ongoing spam/DoS issues. Also, new account creation is currently disabled. We are looking into alternative wiki hosts.

Fixers Available in 2to3

By GregNoel

As part of changing the floor Python version we support to 2.2, we should update our language use to the most modern possible. It's worth doing, since many of the new language features are faster than the equivalent older features. I believe we can automate a great deal of this processing.

Python 3.0 will retire many idioms from older versions of Python, so it provides a program called 2to3 that converts obsolete idioms. It does this by applying a selectable set of "fixers" to code files. Since some of the new idioms actually became available as far back as Python 2.0, the program can be used to update usage for a floor of Python 2.2 by a suitable selection of fixers.

Aside: Strategy for Python 3.0

Since it seems to be a recurring source of confusion, let's take a quick aside to describe the strategy for dealing with Python 3.0.

Supporting two code bases, one for 2.x and one for 3.x, is fraught with peril. There are just too many opportunities for unintentional incompatibilities, either by only updating one code base or by accidentally making different changes.

Instead, the strategy is to create a code base that works on 2.x and can be mechanically translated to work on 3.x. We will maintain the 2.x code base, so all changes will be automatically made to the 3.x code base. This strategy will require some care to make it work, but it is the simplest way to approach the problem.

Many of the fixers do a good job, but punt on complex translations. As a result, they need to be manually tweaked. To deal with this problem, we will try to apply fixers as soon as possible; in other words, this strategy works best if the language we are using is as "modern" as possible. We will use forward-compatibility routines in the compat module to support features that aren't yet available in the base Python version.

Conversely, for those changes which can't be fully automated, we will try to provide backward-compatibility routines in 3.x. Some of these may be difficult, as the Python language itself is different, so the exact code required may need to be a compromise to match what we can support in both cases.

The actual packaging for distribution will probably contain a copy of both code bases. There are several options that can be employed to provide the end-user with the correct package: the installer could identify the system Python in use and install the correct code base, a front-end shim could detect whether the Python version is 2.x or 3.x and choose the correct version to use, or other ingenous solutions.

So the major point of this exercise is to identify which fixers can be applied immediately, with or without compat support, and which fixers must be reserved for the mechanical upgrade to 3.0. If the only fixers for the automated step are basically cosmetic, we will be in good shape. If they will require manual tweaking, life will become more interesting.

Table of Fixers

The table below contains the 47 fixers available in Python 3.0. The Floor column is the important one; it says which fixers may be safely applied with the specified floor. The column can also specify "compat" which means the fixer is safe if the compat module provides forward compatibility or the column can specify "unused" which means it's a feature not used in SCons. As our base Python version moves up, we can apply more of the fixers.

CAVEAT: I am mostly concerned about fixers for a floor of Python 2.2, so this table is incomplete. I'd like help completing the missing entries (or even confirming the entries that are present). If you have knowledge about when fixers are safe to apply or can work on figuring it out empirically, please contact me.

Fixer[1]

Floor

Description

apply*

2.2

apply(fn, arg, kw) ==> (fn)(*arg, **kw)

basestring*

unused

basestring ==> str

buffer[2]*

compat?

buffer(...) ==> memoryview(...)

callable*

2.6

callable(obj) ==> hasattr(obj, '__call__')

dict*

2.2

for ? in keys, items, values:
    dict.?() ==> list(dict.?())
    dict.iter?() ==> iter(dict.?())

except

2.6

except e, t ==> except e as t

exec*

3.0

exec code in ns1, ns2 ==> exec(code, ns1, ns2)

execfile*

Bug#2326

execfile(name, *args) ==> exec(open(name).read(), *args)

filter*

2.2

filter(F, X) ==> list(filter(F, X))

funcattrs

2.6

for ? in closure doc globals name defaults code dict
    f.func_? ==> f.__?__

future*

3.0

remove from __future__ import foo

getcwdu

unused

os.getcwdu() ==> os.getcwd()

has_key*

2.2

d.has_key(k) ==> k in d

idioms[2]*

compat

some type(x) op Type comparisons into isinstance() calls
some list.sort() into sorted(list)
while 1: ==> while True:

import*

2.5[3]

using '.' for local directory imports

imports*

compat [table]

incompatible imports and module references (first stage)

imports2*

incompatible imports and module references (second stage)

input

unused

input(...) ==> eval(input(...))

intern*

compat

intern(s) ==> sys.intern(s)

itertools*

??

itertools.(imap|ifilter|izip) ==> (map|filter|zip)
itertools.ifilterfalse ==> itertools.filterfalse

itertools_imports*

??

imports of itertools.(imap|ifilter|izip|ifilterfalse)

long*

3.0[4]

long ==> int
strips the trailing 'L' or 'l'

map*

2.2

map(lambda ?: <expr>, l) ==> [<expr> for ? in l]
map(F, ...) ==> list(map(F, ...))

metaclass

3.0

__metaclass__ = X ==> (metaclass=X) on classdef

methodattrs

2.6

for ? in func self class:
    method.im_? ==> method.__?__

ne

unused

<> ==> !=

next*

never?

it.next() ==> next(it)

nonzero

2.6

__nonzero__ ==> __bool__

numliterals

2.6

1L ==> 1 and 0755 ==> 0o755

paren

unused

[x for x in 1, 2] ==> [x for x in (1, 2)]

print*

2.6[3]

print statement ==> print() function

raise*

2.2

raise E, V ==> raise E(V)
raise E, V, T ==> raise E(V).with_traceback(T)

raw_input

unused

raw_input(...) ==> input(...)

renames*

compat

sys.maxint ==> sys.maxsize

repr

unused

`xyzzy` ==> repr(xyzzy)

set_literal[2]

3.0

set() calls ==> set literals

standarderror

unused

StandardError ==> Exception

sys_exc

unused

sys.exc_(type|value|traceback) ==> sys.exc_info()[(0|1|2)]

throw

unused

generator.throw(E, V) ==> generator.throw(E(V))

tuple_params

unused

def func(((a, b), c), d): ==> def func(x, d): ((a, b), c) = x
lambda (x, y): x + y ==> lambda t: t[0] + t[1]
lambda (x): x + y ==> lambda x: x + y

types*

2.4?

Changes names from types module into primitives
(e.g., BooleanType ==> bool)
Caution: Does not remove import statement

unicode*

2.6[3]

unicode ==> str
unichr ==> chr
u"..." ==> "..."

urllib*

compat?

imports of urllib which are now incompatible

ws_comma[2]

never?

a ,b ==> a, b
{a :b} ==> {a: b}

xrange

2.2

xrange(...) ==> range(...)
range(...) ==> list(range(...))

xreadlines

unused

for x in f.xreadlines() ==> for x in f

zip

2.2

zip(seq0, seq1, ...) ==> list(zip(seq0, seq1, ...)

[1] The asterisk on the name is a link to the fixer's entry in the "Notes" section below.

[2] Fixer not applied by default and must be explicitly requested.

[3] Feature can be imported from __future__ starting with the specified floor.

Table of Changed Module Names

The imports and imports2 fixers (which must be run in sequence) convert obsolete module names into their new equivalents. Most of the name changes are irrelevant to us; SCons doesn't use them. The few remaining will need to be supported via the compat module. In the table below, the Floor column gives the Python version when the new module is present in the standard release (i.e., when the compat support can be removed).

Caution: these two fixers should be applied to a module only once; note that anydbm and whichdbm are transliterated into dbm, which would be further transliterated into dbm.ndbm in a second invocation.

(((Note that cProfile seems to be missing from the table; it's gone and profile now automatically selects a C-based implementation if it is available.)))

Old name

New name

Floor

Notes

anydbm

dbm

unused

Comments in SConsign.py refer to anydbm

BaseHTTPServer

http.server

unused

CGIHTTPServer

http.server

unused

commands

subprocess

unused

ConfigParser

configparser

unused

Cookie

http.cookies

unused

cookielib

http.cookiejar

unused

copy_reg

copyreg

unused

cPickle

pickle

2.6?

pickle now automatically selects faster version if available; also used in tests

cStringIO

io

2.6?

io now automatically selects faster version if available; see StringIO below. Imported by Node/FS.py but not used.

dbhash

dbm.bsd

unused

dbm

dbm.ndbm

unused

Scanner/Dir.py and script/sconsign.py comments refer to dbm

Dialog

tkinter.dialog

unused

DocXMLRPCServer

xmlrpc.server

unused

dumbdbm

dbm.dumb

unused

Scanner/Dir.py comments refer to dumbdbm

dummy_thread

_dummy_thread

unused

FileDialog

tkinter.filedialog

unused

gdbm

dbm.gnu

unused

httplib

http.client

unused

markupbase

_markupbase

unused

Queue

queue

2.6

repr

reprlib

unused

robotparser

urllib.robotparser

unused

ScrolledText

tkinter.scrolledtext

unused

SimpleDialog

tkinter.simpledialog

unused

SimpleHTTPServer

http.server

unused

SimpleXMLRPCServer

xmlrpc.server

unused

SocketServer

socketserver

unused

StringIO

io

2.6

Used in core, test/option/profile.py, and bin/scons-proc.py

thread

_thread

unused

We use Queue (now queue) but not thread

Tix

tkinter.tix

unused

tkColorChooser

tkinter.colorchooser

unused

tkCommonDialog

tkinter.commondialog

unused

Tkconstants

tkinter.constants

unused

Tkdnd

tkinter.dnd

unused

tkFileDialog

tkinter.filedialog

unused

tkFont

tkinter.font

unused

Tkinter

tkinter

unused

tkMessageBox

tkinter.messagebox

unused

tkSimpleDialog

tkinter.simpledialog

unused

turtle

tkinter.turtle

unused

urlparse

urllib.parse

unused

UserList

collections

2.4?

Used in core, test, QMTest, and bench; may need manual editing

UserString

collections

2.4?

Used in core, test, QMTest, and bench; may need manual editing

whichdbm

dbm

unused

xmlrpclib

xmlrpc.client

unused

_winreg

winreg

2.6

Used in Util.py and script/scons-post-install.py; may need manual editing

__builtin__

builtins

2.6

Used all over compat modules, plus Platform/win32.py, SConfTests.py, and dblite.py. Also test, script, and QMTest. Probably needs to be untangled manually.

Toward a New Floor of Python 2.2

This section is an attempt to keep track of what needs to be done to convert the Python floor version to 2.2, which SCons will (attempt to) for our 2.0 release. The master issue for this conversion is Bug#2345 and the issues to be resolved can be seen in this TODO list.

Notes on Individual Fixers

The apply fixer

Virtually all of the changed content will have to be tweaked manually, at least in the long run, as apply(x, (a,b)) becomes x(*(a,b)), which is correct but inefficient. This is a big job and the plan is to do this fixer individually so that regression tests can be run before and after. The intent is to do this second, after the str_funcs fixer. Bug#2315 has been created to track its application.

The basestring fixer

Only used in the shlex compatibility module, which is no longer needed since the shlex module is standard in Python 2.2. Bug#2324 has been created to deal with the removal of the shlex compatibility module.

The buffer fixer

Only two occurances. The one in the compat module can take care of itself. The other one is in QMTest and will probably require some special-purpose coding to be forward-compatible. Bug#2325 has been created to track the progress of this task.

The callable fixer

Prior to Python 2.6, a simple class (e.g., class c: pass) is callable but has no '__call__' attribute.

The dict fixer

This is described under limitations below; the fixer should be upgraded not to wrap list() around calls in a for loop. Otherwise these will have to be audited by hand, a big job. Bug#2316 has been created to track this upgrade.

Plan to do this fixer individually, not only because the changes will need to be audited but also because there are a lot of changes. Bug#2317 has been created to track the application of this fixer.

The except fixer

No notes at this time

The exec fixer

Nathan Binkert made a great catch by noticing that exec actually accepts a tuple as the first argument, so
    a = { 'x' : 1 }
    b = ()
    c = ('print x', a, b)
    exec(c)
works and prints '1'. This looks a lot like an exec function.

Unfortunately, this sequence gets an error in Python 3.0, so I'm not sure what it gets us. We would still need an automated translation to convert from the 2.x code base we will be maintaining to a 3.x-compatible version. I'm inclined to think it will easier to keep the exec statements as-is and use the existing fixer.

The execfile fixer

Uses of 'execfile()' should be replaced with a backward-compatible cross-platform equivalent using the exec statement. That is,
    execfile(name [, global [, local]])
should be manually converted to
    exec open(name).read().replace('\r', '\n') [in global [, local]]
so that when the time comes, the exec fixer will correctly convert to a platform-independent implementation. Bug#2326 has been created to track this conversion.

The filter fixer

This fixer misses possible conversions into list comprehensions. To make our code compatible all the way back to Python 1.5.2, many of our lambda functions have extra arguments simply to copy variables in from the enclosing scope. Identifying these cases automatically may be more effort than making the conversions manually.

The converter also seems to convert some calls that are in iterator contexts; see limitations below. These could be detected automatically, but there appear to be only a few, so correcting them manually should not be too much effort.

But see map; many of the uses of filter() are in conjunction with nearby map() calls. These can often be combined into a for loop that is faster. This identification and rewriting will have to be done manually.

So plan to do this fixer individually. Note that it must be done before the itertools converters, as they can produce new filter() calls. Bug#2318 has been created to track the application of this fixer.

The funcattrs fixer

No notes at this time

The future fixer

Although it would be possible to apply this fixer immediately, there will be more from __future__ statements added in planned enhancements (see import, print, and unicode), so just leave the import statements in for now and apply the fixer as a part of the transition to Python 3.0.

The getcwdu fixer

No notes at this time

The has_key fixer

This fixer makes a lot of changes; enough so that the fixer should be applied individually, with regression tests before and after. Bug#2319 has been created to track the application of this fixer.

The idioms fixer

Requires compat support to provide a sorted() global function. Bug#2327 has been created to track the development of this enhancement to to the compat support.

The conversion for sorted can botch the indentation of the line after the changed region (see limitations below. Bug#2328 has been created to track the upgrade to the fixer.

This fixer makes a lot of changes; it should be applied individually. Bug#2329 has been created to track the application of this fixer.

The import fixer

Requires from __future__ import absolute_import which is first available in Python 2.5.

The imports and imports2 fixers

Requires compat support to implement import pickle, choosing cPickle if it is available. Bug#2330 has been created to track the development of this enhancement to the compat support.

Requires compat support to implement import profile, choosing cProfile if it is available. Bug#2331 has been created to track the development of this enhancement to the compat support.

Requires compat support to implement import io containing cStringIO.StringIO (as StringIO) if it is available, otherwise StringIO.StringIO. Bug#2332 has been created to track the development of this enhancement to the compat support.

Requires compat support to implement import queue as selecting the Queue module. Bug#2333 has been created to track the development of this enhancement to the compat support.

Requires compat support to implement import collections containing UserList.UserList and UserString.UserString. Bug#2334 has been created to track the development of this enhancement to the compat support.

Requires compat support to implement import winreg (was _winreg) Bug#2335 has been created to track the development of this enhancement to the compat support.

Requires compat support to implement import builtins (was __builtin__) Bug#2336 has been created to track the development of this enhancement to the compat support.

These fixers should be applied individually. References to cProfile will have to be fixed manually. Bug#2337 has been created to track the application of this fixer.

After this fixer is applied, tasks should be undertaken to convert base classes of collections.UserList to list and base classes of collections.UserString to str. Bug#2338 has been created for the former and Bug#2339 has been created for the latter.

The input fixer

No notes at this time

The intern fixer

Requires compat support. Bug#2340 has been created to track the development of this enhancement to the compat support.

The itertools and itertools_imports fixers

The itertools and itertools_imports fixers must be run after the map, filter, and zip fixers. Bug#2341 has been created as a placeholder to specify the ordering.

The long fixer

SCons uses one long constant (in the subprocess compatibility module), which can be retired, since the Python 2.x parser now creates a long type if the input value is too large for an int type. Bug#2342 has been created to track the removal of the L on the constant.

On the other hand, SCons uses type(1L) and long to get a type that it assumes is distinct from int. Simulating this in a forward-compatible way is a distinct challenge; it's likely that the remainder of this fixer cannot be applied any earlier than 2.6, and maybe not until 3.0. See the notes about type changes below.

The map fixer

The map fixer makes a massive number of changes. There's no doubt that this fixer will be applied individually, with regression tests before and after. Bug#2321 has been created to track the application of this filter.

But see also filter; cases where several map() and filter() functions occur together can often be combined into a for loop that is faster. This identification and rewriting will have to be done manually.

Moreover, many map() calls could have been converted to list comprehensions, but the map fixer couldn't recognize them because the lambda function had more than one parameter (where the additional parameters were simply to pass in values from the current scope). If the fixer cannot be modified to convert these automatically (or simply mark them for manual tweaking), each of the changes will have to be examined and corrected (or marked) manually. Bug#2320 has been created to track this upgrade.

The metaclass fixer

No notes at this time

The methodattrs fixer

No notes at this time

The ne fixer

No notes at this time

The next fixer

This fixer attempts to convert the next method in the Walker class as if it were an iterator. Unfortunately, it's not: instead of rasing a StopIteration exception when the iteration is done, it returns None. As a result, we should never apply this fixer.

Unless, of course, we get sone code that uses generators and expects the next() method to be called by the system. In that case, I don't know what to suggest.

The nonzero fixer

No notes at this time

The numliterals fixer

No notes at this time

The paren fixer

No notes at this time

The print fixer

Requires from __future__ import print_function which is first available in Python 2.6.

The raise fixer

If the second argument is an exception instance, this fixer will convert it incorrectly. It does so in at least one case, in Taskmaster.py, where a three-argument raise is used. This one will have to be converted manually. Moreover, the conversion uses with_traceback() on an exception instance, which doesn't seem to exist until Python 2.6. Bug#2322 has been created to track the creation of a forward-compatible solution.

This fixer also finds quite a number of raising string exceptions. These will have to be resolved one at a time to make sure that anyplace they are caught, the string is retrieved correctly. There's only one case in core code (in Node/FS.py); the others are all in tests, so they're probably being caught by the test framework in one place.

The changes proposed by this fixer are quite large, and there's the manual effort, so this fixer will have to be applied individually. Bug#2323 has been created to track the application of this fixer.

The raw_input fixer

No notes at this time

The renames fixer

Requires compat support to move sys.maxint to sys.maxsize.

The repr fixer

No notes at this time

The set_literal fixer

No notes at this time

The standarderror fixer

No notes at this time

The str_funcs fixer

This fixer is not one present in the Python distribution. We must write it. Bug#2308 has been created to track its development.

There are over a thousand places to be fixed. That's one of the largest set of changes, so apply the fixer individually, with regression tests before and after. Bug#2314 has been created to track the application of this fixer.

The sys_exc fixer

No notes at this time

The throw fixer

No notes at this time

The tuple_params fixer

No notes at this time

The types fixer

Many of the values formerly in the types module have gone away, to be replaced by global type namess (e.g., types.DictType becomes dict). Since not all names in the types module have been replaced, the import for the module is not removed.

Most of these changes can be applied for Python 2.2, but not all: there is no types.BooleanType or primitive bool type, types.LongType is converted to int, and types.UnicodeType is converted to str.

It's possible that all of the names converted by this fixer could be converted to standard primitive types manually, in effect making this fixer unused. See the type changes section below.

The unicode fixer

Requires from __future__ import unicode_literals which is first available in Python 2.6.

The urllib fixer

The urllib structure has been reorganized. It's used by bin/scons-test.py to get test files from the network. Bug#2344 has been created to track making this compatible no matter which structure is present.

The ws_comma fixer

No notes at this time

The xrange fixer

No notes at this time

The xreadlines fixer

No notes at this time

The zip fixer

No notes at this time

Table of Residual Fixers

This table contains the fixers we are not applying during the conversion to a floor of Python 2.2. The first column is fixers that have no effect on the current code base, but if we ever incorporate the related features, we will have to determine whether to support them with forward compatibility (and remove the fixer from the table) or apply the fixer at a later point (and move the fixer to a different column). The remaining columns are when the Python language was extended to cover the specified feature; when our floor becomes that version, we can apply the fixer. Another way to say it is that the fixers in the remaining columns are the ones we need to apply when generating the Python 3.x flavor of our code for distribution.

unused

2.3

2.4

2.5

2.6

3.0

getcwdu
input
paren
raw_input
standarderror
sys_exc
throw
xreadlines

(none)

(none)

import

except
funcattrs
methodattrs
nonzero
numliterals
print
unicode

basestring
exec
future
long
metaclass
set_literal
types

Type Changes

The objective is to provide a platform that at least appears uniform over all the Python versions we are supporting. In some cases, this will be by providing forward compatiblity within 2.x and in others it will be by providing backward compatibility within 3.x. As long as the base code works the same, it doesn't matter if it's a bit inefficient in some cases.

The fixers in question are types, unicode, long, basestring, buffer (for memoryview), xxx

The process we will use is to maintain a code base for whatever Python 2.x versions we are supporting, with forward compatibility so that the language we are using is the most modern possible. For Python 3.x, we will run a selected set of fixers over a copy of the Python 2.x code base and provide routines for backward compatibility. The saving grace may be that we can create special fixers to apply custom changes that will minimize the distance between these two.

We'll probably have to apply most of the changes described in this section manually, maybe with some fixer help.

In the table below, a type in the first column is marked with an asterisk if it is not present in Python 2.2 and a plus if it is present but as a function instead of a type. A typename in the second column is marked with an asterisk if it appears in the SCons code and with a plus if it appears in comments. Here's a starter table for the types that concern us:

Types that mostly should "just work"

Type

TypeName

Notes

bool*

BooleanType*

The bool type will have to be simulated for Python 2.2. Note that there is already compat support for bool but it's just a renamed int. Python 2.2 will allow a more accurate simulation; see PEP 285.

The uses of the values within the types module will need to be validated, but a quick scan suggests that there should be no problem with using the global types; the only reason they weren't used before was for backward compatibility with Python 1.5.2.

complex

ComplexType

dict

DictType*

dict

DictionaryType*

float

FloatType

list

ListType*

object

ObjectType

range+

XRangeType

slice+

SliceType

tuple

TupleType

type

ClassType

type

TypeType*

Unification of int and long

Type

TypeName

Notes

int

IntType*

The long fixer converts uses of the long type to int. Either the fixer can be run or we can provide a long type in Python 3.x that's really int.

The types fixer converts LongType to int so any instances of this variable will have to be changed manually (there don't appear to be any).

long

LongType

Strings and memory access

Type

TypeName

Notes

basestring*

BaseStringTypes*

In Python 2.2, the basestring type isn't present yet, so it will have to be simulated (or coded around?). In Python 3.0, all strings are Unicode, there's a new bytes type for raw memory access, and the basestring type goes away. String constants may or may not be Unicode. The u prefix to specify a Unicode string constant is either required or forbidden. This is the nastiest problem; I just don't know how to write compatible code over this entire range.

The types fixer converts UnicodeType to str, so this change will have to be done manually. The types fixer should convert the remaining references.

There will need a strategy for supporting the types as long as our floor is a 2.x Python version. Nathan Binkert is looking at this.

memoryview*

BufferType

bytes*

StringType*

str

StringTypes*

unicode

UnicodeType*

Oddball types

Type

TypeName

Notes

io.IOBase

FileType

I don't think we use any of these...

type(Ellipsis)

EllipsisType

type(None)

NoneType+

type(NotImplemented)

NotImplementedType

Known 2to3 Limitations

2to3 doesn't convert everything, nor are all the conversions perfect. In general, the transformations need to be audited by a human being to verify that the intent was correct. This section identifies the limitations encountered so far.

No fixer for string module functions

Functions which used to be in the string module (i.e., invoked as string.name(str,...)) are now invoked as string methods (i.e., invoked as str.name(...)). We have many hundreds (perhaps thousands) of these occurrences and there's no fixer to convert them.

Bug#2308 covers the development of a new fixer. There are several fixers that do similar things, so it should be possible to base this code on an existing fixer, which should simplify development. Once it's written, we will offer it to the Python folks to see if they want to adopt it.

The sorted() conversion is incorrect

The idioms fixer converts a multi-line idiom into a single line. Unfortunately, if the idiom is the last thing in an indented scope, it will pick up the first line of the next scope and indent it. That is, this sequence
    if do_sort:
        keys = d.keys()
        keys = keys.sort()
    l = []
is converted into this sequence
    if do_sort:
        keys = sorted(d.keys())
        l = []
Note that the indentation of the last line is incorrect and that this is a silent semantic change.

I imagine what's happening is that the sorted list is often used immediately in the next statement and the variable containing the sorted list (keys in the example) is not used anywhere else. In this case, folding the calculation into the next statement will eliminate the unneeded variable. However, the lookahead fails to deal with the indentation.

The apply converter creates inefficient code

The apply converter doesn't look at the arguments it is converting and can create inefficient code. For example,
    apply(fn, (a,b))
is converted to
    fn(*(a,b))
which is correct, but inefficient. Since the argument is a tuple, the fixer could expand it immediately, giving a conversion of
    fn(a,b)
which is both correct and efficient.

A similar thing happens with the third argument if it's a dict constant, but I'd imagine this pattern is uncommon enough that it could be handled manually. That is,
    apply(fn, (a,b), { 'param' : 'value' })
could be converted to
    fn(a, b, param = 'value')

Convert more map() calls into list comprehensions

The map fixer could convert more map(lambda ..., V) expressions. In particular,
    map(lambda X, a=a,b=b: <expr>, V)
can be converted into
    [<expr> for X in V]

Note that as long as the keyword arguments are exactly of the form V=V then all they are doing is copying values from the enclosing scope into the lambda scope. Since a list comprehension does not introduce a new scope, these arguments can be safely ignored in the conversion.

Forcing a list in an iteratable context

Quite a number of functions and methods that used to return a list will now return an iterable. One of the implications of this is that in cases where code expects, say, dict.keys() to return a list could be rudely surprised when it returned an iterable. The solution is to wrap the value in a list(...) so that the iterable is evaluated. This is the solution adopted for a number of fixers.

However, there are cases where returning an iterable is exactly the right thing to do. For example,
    for k in dict.keys():
        ...
need not be converted. At least one fixer (map) does conversions this way; all of them should.

The ones I've found with this problem are dict, xrange, and probably zip, but there may be others.

The execfile conversion should canonicalize line endings

The conversion performed by the execfile fixer is not backward compatible. The execfile() function deals correctly with line endings of any sort, whether <LF> (UNIX, Linux, ...), <CR> (Mac), or <CR><LF> (DOS). I don't believe this was intentional (it's certainly not documented); it's because the code implementing execfile() uses the C library fopen() function with a "r" (rather than "rb") parameter, which causes newlines to be canonicalized.

Instead of
    open(name).read()
the execfile fixer should use
    open(name).read().replace('\r', '\n')
to get backward-compatible cross-platform ability.

PythonFixers (last edited 2010-03-28 08:28:05 by ip68-7-77-81)