datetime.utcfromtimestamp rounds results incorrectly

@tim-one

BPO	23517
Nosy	@tim-one, @mdickinson, @abalkin, @vstinner, @larryhastings, @bitdancer, @serhiy-storchaka
Files	round_half_even_py34.patch

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = 'https://github.com/abalkin'
closed_at = <Date 2015-09-18.13:11:31.154>
created_at = <Date 2015-02-24.22:38:42.616>
labels = ['type-bug', 'library']
title = 'datetime.utcfromtimestamp rounds results incorrectly'
updated_at = <Date 2015-09-18.13:11:31.153>
user = 'https://bugs.python.org/tbarbugli'

bugs.python.org fields:

activity = <Date 2015-09-18.13:11:31.153>
actor = 'vstinner'
assignee = 'belopolsky'
closed = True
closed_date = <Date 2015-09-18.13:11:31.154>
closer = 'vstinner'
components = ['Library (Lib)']
creation = <Date 2015-02-24.22:38:42.616>
creator = 'tbarbugli'
dependencies = []
files = ['40414']
hgrepos = []
issue_num = 23517
keywords = ['patch', '3.3regression']
message_count = 100.0
messages = ['236552', '236553', '236577', '236580', '236581', '236585', '236587', '236589', '236597', '236607', '236608', '236609', '236610', '246049', '246073', '246097', '246104', '246121', '246284', '246306', '246307', '246334', '248942', '249282', '249300', '249301', '249303', '249304', '249307', '249308', '249309', '249519', '249520', '249521', '249525', '249528', '249530', '249532', '249539', '249569', '249611', '249728', '249744', '249771', '249777', '249778', '249779', '249786', '249787', '249788', '249791', '249796', '249798', '249825', '249827', '249834', '249835', '249836', '249841', '249842', '249844', '249845', '249847', '249848', '249849', '249850', '249851', '249853', '249856', '249875', '249879', '249885', '249886', '249898', '249909', '249910', '249925', '249926', '250043', '250047', '250048', '250049', '250066', '250075', '250083', '250101', '250103', '250104', '250107', '250108', '250113', '250259', '250265', '250268', '250269', '250270', '250304', '250405', '250973', '250974']
nosy_count = 13.0
nosy_names = ['tim.peters', 'mark.dickinson', 'belopolsky', 'vstinner', 'larry', 'r.david.murray', 'aconrad', 'BreamoreBoy', 'vivanov', 'python-dev', 'serhiy.storchaka', 'tbarbugli', 'trcarden']
pr_nums = []
priority = 'normal'
resolution = 'fixed'
stage = 'commit review'
status = 'closed'
superseder = None
type = 'behavior'
url = 'https://bugs.python.org/issue23517'
versions = ['Python 3.4', 'Python 3.5', 'Python 3.6']

Hi,

I am porting a library from python 2.7 to 3.4 and I noticed that the behaviour of datetime.utcfromtimestamp is not consistent between the two versions.

For example on python 2.7.5
datetime.utcfromtimestamp(1424817268.274)
returns a datetime with 274000 microseconds

the same code in python 3.4 returns a datetime with 273999 microseconds.

This seems to have changed in 3.3 (versions up to 3.2 return 274000).

Most likely this was a rounding fix (ie: not a bug), but hopefully Alexander will know for sure.

Let me dig up the history, but this does not look like correct rounding to me:

>>> datetime.utcfromtimestamp(1424817268.274)
datetime.datetime(2015, 2, 24, 22, 34, 28, 273999)
>>> decimal.Decimal(1424817268.274)
Decimal('1424817268.2739999294281005859375')

It looks like it was an intentional change. See bpo-14180 (changeset 75590:1e9cc1a03365).

I am not sure what the motivation was. Note that this change made utcfromtimestamp(t) different from datetime(1970,1,1) + timedelta(seconds=t).

Victor's motivation for the change was (msg154811):

"""
I chose this rounding method because it is the method used by int(float) and int(time.time()) is a common in programs (more than round(time.time()). Rounding towards zero avoids also producing timestamps in the future.
"""

I recall the earlier discussions of rounding in the datetime module and Mark's explanation that rounding up is fine as long as ordering is preserved. i.e. for x < y round(x) <= round(y).

There are cases when producing times in the future are problematic, for example UNIX make really dislikes when file timestamps are in the future, but if this was the main motivation - rounding towards -infinity would be more appropriate.

In any case, as long as we have the following in the datetime module documentation, I think this behavior is a bug:

"""
On the POSIX compliant platforms, utcfromtimestamp(timestamp) is equivalent to the following expression:

datetime(1970, 1, 1) + timedelta(seconds=timestamp)
"""

>>> timestamp = 1424817268.274
>>> datetime.utcfromtimestamp(timestamp) == datetime(1970, 1, 1) + timedelta(seconds=timestamp)
False

I started a large change set to support nanoseconds in the C "pytime" API: see the issue bpo-22117. While working on this change, I noticed that the rounding mode of datetime is currently wrong. Extract of a private patch:

typedef enum {
/* Round towards zero. */
_PyTime_ROUND_DOWN=0,
/* Round away from zero.
For example, used for timeout to wait "at least" N seconds. */
_PyTime_ROUND_UP=1,
/* Round towards minus infinity (-inf).
For example, used for the system clock with UNIX epoch (time_t). */
_PyTime_ROUND_FLOOR=2
} _PyTime_round_t;

I changed Modules/_datetimemodule.c to use _PyTime_ROUND_FLOOR, instead of _PyTime_ROUND_DOWN.

I noticed that the rounding mode of datetime is currently wrong.

What do you mean by "currently"? What versions of python have it wrong?

Victor,

Would you consider going back to round to nearest? Mark and I put in a lot of effort to get the rounding in the datetime module right. (See for example, bpo-8860.)

Sub-microsecond timesources are still rare and users who work with such should avoid FP timestamps in any case. On the other hand, double precision timestamps are adequate for microsecond resolution now and in the next few decades.

Timestamps like OP's (sec=1424817268, us=274000) should not change when converted to double and back. IMO, the following behavior is a bug.

>>> dt = datetime(2015, 2, 24, 22, 34, 28, 274000)
>>> datetime.utcfromtimestamp(dt.timestamp())
datetime.datetime(2015, 2, 25, 3, 34, 28, 273999)

Would you consider going back to round to nearest?

I don't understand "nearest". I prefer to use names of decimal rounding modes:
https://docs.python.org/dev/library/decimal.html#rounding-modes

In my local patch, I'm using ROUND_FLOOR in _decimal: "Round towards -Infinity."

Mark and I put in a lot of effort to get the rounding in the datetime module right. (See for example, bpo-8860.)

I'm unable right now to say which rounding mode should be used in the decimal module. But it's important to use the same rounding mode for all similar operations. For example, time.time() and datetime.datetime.now() should have the same rounding method (bad example, time.time() returns a float, which doesn't round the result).

For example, in my local patch, I'm using ROUND_FLOOR for:

datetime.date.fromtimestamp()
datetime.datetime.fromtimestamp()
datetime.datetime.now()
datetime.datetime.utcnow()
os.utime()
time.clock_settime()
time.gmtime()
time.localtime()
time.ctime()

Note: the Python implementation of datetime uses time.localtime() and time.gmtime() for fromtimestamp(), so these functions should also have the same rounding method.

What do you mean by "currently"? What versions of python have it wrong?

I search for "ROUND" in Modules/_datetimemodule.c: in the Python development branch (default), I found _PyTime_ROUND_DOWN (Round towards zero). Since a bug was reported, I understand that it's not the good rounding method?

I don't understand "nearest".

Sorry for using loose terms. I was hoping the in the context of "going back", it would be clear.

I believe the correct mode is "ROUND_HALF_EVEN". This is the mode used by the builtin round() function:

>>> round(0.5)
0
>>> round(1.5)
2

For example, in my local patch, I'm using ROUND_FLOOR for:

datetime.date.fromtimestamp()

datetime.datetime.fromtimestamp()

These should use ROUND_HALF_EVEN

datetime.datetime.now()

datetime.datetime.utcnow()

These should not involve floating point arithmetics, but when converting from nanoseconds to microseconds, you should round to nearest 1000 ns with 500 ns ties resolved to even number of microseconds.

os.utime()

This takes nanoseconds as an optional argument. Passing floats in times should probably be deprecated. In any case, here you would be rounding floats to nanoseconds and what you do with 0.5 nanoseconds is less important because in most cases they are not even representable as floats.

time.clock_settime()

Is this a new method? I don't see it in 3.5.0a1.

time.gmtime()

This should be fixed

>>> time.gmtime(1.999999999).tm_sec
1

is really bad and

>>> time.gmtime(-1.999999999)[:6]
(1969, 12, 31, 23, 59, 59)

is probably even worse.

time.localtime()

time.ctime()

Same story as in time.gmtime.

All you need to do is to revert a few chunks of 1e9cc1a03365 where the regression was introduced.

Hum, please don't revert this change. I spent a lot of days to write pytime.c/.h.

My patches add more unit tests to datetime, to test the exact rounding mode.

Well, for now I assume it really truly genuinely isn't going in 3.5.0. I suppose we can debate about 3.4.x and 3.5.1 later, once we have a fix everybody is happy with.

The rounding mode of microseconds only impact a very few people. The rounding mode changed in 3.3, again in 3.4 and again in 3.5. This issue is the first bug report. So I think it's fine to modify 3.4.x and 3.5.x.

Victor> please don't revert this change.

I did not suggest reverting the entire commit. The change that affects fromdatetime() is just

```
   us = round(frac * 1e6)
```

+ us = int(frac * 1e6)

in datetime.py. It is probably more involved in _datetimemodule.c, but cannot be that bad. You can still keep pytime.c/.h.

FWIW, count me as +1 on roundTiesToEven, mostly just for consistency. It's easier to remember that pretty much everything in Python 3 does round-ties-to-even (round function, string formatting, float literal evaluations, int-to-float conversion, Fraction-to-float conversion, ...) than to have to remember that there's a handful of corner cases where we do roundTiesToAway instead.

New changeset d0bb896f9b14 by Victor Stinner in branch 'default':
Revert change 0eb8c182131e:
https://hg.python.org/cpython/rev/d0bb896f9b14

New changeset 171d5590ebc3 by Victor Stinner in branch 'default':
Issue bpo-23517: fromtimestamp() and utcfromtimestamp() methods of
https://hg.python.org/cpython/rev/171d5590ebc3

I wish people wouldn't remove old patches. There's no harm in leaving them, and it may be a useful historical record.

I modified Python 3.6 to use ROUND_HALF_EVEN rounding mode in datetime.timedelta, datetime.datetime.fromtimestamp(), datetime.datetime.utcfromtimestamp().

round_half_even_py34.patch: Backport this change to Python 3.4. I tried to write a minimal patch only changing datetime, not pytime.

I wish people wouldn't remove old patches. There's no harm in leaving them, and it may be a useful historical record.

It became hard to read this issue, it has a long history. My old patches for Python 3.4 were for ROUND_HALF_UP, but it was then decided to use ROUND_HALF_EVEN. Technically, patches are not "removed" but "unlinked" from the issue: you can get from the history at the bottom of this page.

Given Victor's reluctance to get this in to 3.5.0, this can't even be marked as a "deferred blocker" anymore. Demoting to normal priority.

Alexander: can you please review attached round_half_even_py34.patch? It's the minimum patch to fix datetime/timedelta rounding for Python 3.4 (and 3.5).

New changeset ee1cf1b188d2 by Victor Stinner in branch '3.4':
Issue bpo-23517: Fix rounding in fromtimestamp() and utcfromtimestamp() methods
https://hg.python.org/cpython/rev/ee1cf1b188d2

Ok, I fixed Python 3.4 and 3.5 too: fromtimestamp() and utcfromtimestamp() now uses also ROUND_HALF_EVEN rounding mode, as timedelta constructor. I explained in Misc/NEWS that the change was made to respect the property:

(datetime(1970,1,1) + timedelta(seconds=t)) == datetime.utcfromtimestamp(t)

Thanks Alexander Belopolsky & Tim Peters for your explanation on rounding. It's not a simple problem :-)

Thanks Tommaso Barbugli for the bug report: it's now fixed on all (maintained) Python 3 versions: 3.4, 3.5 and 3.6. The fix will be part of Python 3.5.1.

Navigation Menu

Repository navigation

datetime.utcfromtimestamp rounds results incorrectly #67705

80 remaining items

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

datetime.utcfromtimestamp rounds results incorrectly #67705

Description

Activity

tbarbugli commented on Feb 24, 2015

ethanfurman commented on Feb 24, 2015

bitdancer commented on Feb 25, 2015

abalkin commented on Feb 25, 2015

abalkin commented on Feb 25, 2015

abalkin commented on Feb 25, 2015

vstinner commented on Feb 25, 2015

abalkin commented on Feb 25, 2015

abalkin commented on Feb 25, 2015

vstinner commented on Feb 25, 2015

vstinner commented on Feb 25, 2015

abalkin commented on Feb 25, 2015

abalkin commented on Feb 25, 2015

80 remaining items

vstinner commented on Sep 7, 2015

abalkin commented on Sep 7, 2015

mdickinson commented on Sep 7, 2015

python-dev commented on Sep 8, 2015

python-dev commented on Sep 8, 2015

larryhastings commented on Sep 9, 2015

vstinner commented on Sep 9, 2015

vstinner commented on Sep 9, 2015

larryhastings commented on Sep 9, 2015

vstinner commented on Sep 10, 2015

python-dev commented on Sep 18, 2015

vstinner commented on Sep 18, 2015

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions