Go: sizeof: new package with constants for Int, Uint, Uintptr, Int8, Float64, etc

[robpike]

See the tail of the (closed) proposal #5602. There I suggested that instead of making unsafe.Sizeof a safe thing, which it still kinda isn't in general, we solve 90% of the problem by add easy-to-use constants to a safe package, such as:

package reflect

const (
  SizeofInt = <size on this machine>
  SizeofUint = SizeofInt
  SizeofBool = 1
  SizeofFloat64 = 8
  //...
)

These would be defined for primitive types only, not slices, maps, etc. You'd still need the unsafe package to handle them.

[mvdan]

The closest thing we have right now is bits.UintSize, which would be equivalent to the proposed reflect.SizeofUint: https://golang.org/pkg/math/bits/#pkg-constants

Have you considered reversing the names, like IntSize and UintSize? They'd be a bit shorter and easier to remember, I think.

I also wonder if we could remove bits.UintSize in the future, if all these constants are available in the reflect package.

[rogpeppe]

I think I'm generally -1 on this proposal. I can't see the sizes of any numeric types other than int, uint and uintptr changing in the future. For example, is it remotely possible that uint64 ever gets represented with something other than 8 bytes? I suspect not.

Instead I'd suggest adding one constant to math/bits:

// UintPtrSize is the size of a uintptr in bits.
const UintPtrSize = uintPtrSize

everything else is OK without constants AFAICS.

[rogpeppe]

we solve 90% of the problem by add easy-to-use constants to a safe package

I'd be interested to hear more about what the problem actually is. What's the use case for knowing representation sizes (as opposed to numeric bit sizes) unless it's to go along with other unsafe operations?

[rsc]

@rogpeppe it would be weird to add UintptrSize and not have any routines involving uintptrs in math/bits. (Also it's Uintptr because the lowercase is not uintPtr.)

Adding new constants to reflect does not seem nearly as nice or general as the current unsafe.Sizeof. If we can't find a better home for that language-level constant mechanism, probably its best to just leave it where it is and not have two.

[andybons]

@robpike have your thoughts on this issue changed at all?

[robpike]

I still think it's wrong that the library says that to find the size of int is unsafe: calling unsafe.Sizeof(int) just isn't unsafe. You can compute its size, but it's really tricky and hard to work out from first principles (see the constants blog post for the mechanism).

I still think a handful of constants in someplace safe is a friendlier solution than using unsafe.Sizeof for this subset of its use.

[rsc]

we solve 90% of the problem by add easy-to-use constants to a safe package, such as:

package reflect

const (
  SizeofInt = <size on this machine>
  SizeofUint = SizeofInt
  SizeofBool = 1
  SizeofFloat64 = 8
  //...
)

These would be defined for primitive types only, not slices, maps, etc.

Nearly all the primitive types are already sized (uint8, float64 etc). The only unsized types are int, uint, uintptr, and bool, right? I am skeptical that there is so much code doing unsafe.Sizeof(true) to find out how big a bool is. Checking the size of an int is certainly common. bits.UintSize is the size of an int or uint in bits (but its bits).

Maybe we could add to reflect just:

const (
    IntSize = ...
    PtrSize = ...
)

Maybe IntSize belongs in math?
Math is mostly about floats, but it does have MaxInt8 etc.

Maybe PtrSize belongs in runtime?
It would be unfortunate for code to import reflect just to find out how big an int is (not as unfortunate as importing unsafe but still unfortunate).

[andybons]

@robpike what do you think?

[robpike]

Sounds fine. I'd be fine with them all in reflect - it's a reflective operation to ask a question about a type, and reflect is part of everything that formats values so it's not a burden to the binary - but am open to other options. Maybe even a standalone tiny package of constants.

[robpike]

They could even go into builtin...

[smasher164]

I'm not sure if the ship has sailed on the package to which these belong, but doesn't it seem inconsistent to already have the bit size for a builtin in math/bits (uint), and still declare the bit sizes for int and uintptr in separate packages? I see math/bits as the singular location containing operations and constants to deal with integral types at the bit-level. Declaring these in separate packages would only serve to confuse users and add to the number of dependencies.

[rsc]

Summary of discussion so far:

I think from the original proposal we're down to IntSize, UintSize, and PtrSize.

For location, math/bits doesn't seem perfect but bits.UintSize already exists.
We could put them elsewhere but no other package seems better.
We don't want to import reflect just for sizes.
Using runtime (next to runtime.GOARCH) would be defensible but maybe not better than math/bits.
Math would be a bit odd since there is nothing about unsized integers in its API today.
And 'builtin' is not a real package at the moment.
Maybe math/bits is the best place.

bits.UintSize already exists.
Do we need to add bits.IntSize too?

bits has nothing to do with pointers, so bits.PtrSize would be a mistake.
Probably reflect.PtrSize is the answer there? Reflect is all about pointers.

So it looks like the open question are:

1. Do we add bits.IntSize = UintSize ?
2. Do we add reflect.PtrSize = ... ?

[robpike]

If you put UintSize into bits, I'd put PtrSize there too so they're in one place.

[rsc]

@robpike, sorry for the delayed response. I am worried about the message putting PtrSize into math/bits sends. Math/bits is entirely arithmetic operations on integers, hardly any of which are appropriate or even safe to do on Go pointers (maybe TrailingZeros). It would stick out in the API docs ("what is this doing here?"). In contrast, reflect has lots of pointer stuff.

[robpike]

@rsc, I think a comment can cover that well enough, and it sends a nice message to put all the Size constants in one place.

[smasher164]

Just to clarify, is the current consensus to change the constant declarations in math/bits to something like the following?

const uintSize = 32 << (^uint(0) >> 32 & 1) // 32 or 64

const ptrSize = 32 << (^uintptr(0) >> 63)

const (
    // UintSize is the size of a uint in bits.
    UintSize = uintSize

    // IntSize is the size of an int in bits.
    IntSize = UintSize

    // PtrSize is the size of a uintptr in bits.
    PtrSize = ptrSize
)

[rsc]

@smasher164, I think the consensus is to add IntSize = UintSize to math/bits.

I still have serious reservations about adding PtrSize anywhere but reflect.
There is not a mention of a pointer anywhere in math/bits and I still believe it's important not to even give a hint that bit tricks on pointer representations is a good idea, which the name bits.PtrSize would.

Looking through the conversation history I am not sure why we started talking about the number of bits in a pointer at all.

@rogpeppe, you suggested adding UintptrSize to math/bits.
What was the motivation behind that suggestion?
When would you use it?

Ian points out that UintptrSize == UintSize now on all supported architectures but of course we might not want to assume that for all time. But still, why care about UintptrSize at all?

[rsc]

Ping @rogpeppe or anyone else for why we need UintptrSize at all.

[rogpeppe]

@rsc For consistency. Why should one numeric type with implementation-defined size be different from the others?

[robpike]

I will reiterate a point I made above: Much of the argument here is a consequence of the decision to put this in math/bits, where I don't think it belongs. Why not create a new package, called say size, and put them there? This also opens a place to put helpful functions to safely compute things like the size of an array or struct or slice or possibly? map.

[zephyrtronium]

I have used the size of uintptr in a set type, effectively map[*T]struct{} but with a Reset method to quickly reuse the same memory. Keys were uintptr, coming from e.g. reflect.ValueOf(x).Pointer(). I used a scrambler of the form A*key+C to try to spread out which bits were used for nearby keys; the values of A and C depend on the size of uintptr. I found after pushing that my original implementation didn't compile specifically on 32-bit targets, because the A and C constants overloaded uintptr there; the final implementation uses apparently nonsense type conversions between uint64 and uintptr on a branch that is dead in the cases that require them to be there.

Later, after much more reading, I got the feeling that garbage collection could move objects while I was using such a set, so I made the keys be just a new field on T; the only requirement for them to remain uintptr was that I had already published the set type with that API.

The overall experience suggests to me that anyone who wants to use the size of uintptr probably should know how to find it (although I'd probably say the same is true of bits.UintSize in the first place, so that isn't a real argument). Also, at least in my case, it was easier to do if ^uintptr(0) != 0xffffffff than to care about the exact size.

[rsc]

Talked to @robpike about this. Given the trouble with finding an existing home for these, he suggests a new package:

// Sizeof defines the sizes of basic Go types in bytes.
package sizeof

const (
    Bool = 1
    Int8 = 1
    Uint8 = 1
    ...
    Int = ...
    Uint = ...
    Uintptr = ...
)

These could almost be defined as, for example, Int = unsafe.Sizeof(int(0)), but that would result in Int being a uintptr, not an untyped constant.

Having a separate package eliminates all the problems with none of the existing packages being right.

Retitling for this new idea.

[rsc]

Does anyone have any objections to this new package?
The idea is that it would be import "sizeof".

[rsc]

Based on the reactions above, it sounds like we've converged to a likely accept for the new sizeof package.

[rsc]

No change in consensus, so accepted.

[carnott-snap]

Considering the help wanted flag and the fact that the discussion has stalled, I thought I would offer an implementation for review: (I took the liberty of adding Interface, Map, Slice, Channel, and Function, since they are constant size pointer like types, but left off Array and Struct since they are variable size value like types.)

// Package sizeof defines the sizes of basic Go types in bytes.
package sizeof

const (
    Bool       = 1
    Byte       = Uint8
    Channel    = Uintptr
    Complex128 = 16
    Complex64  = 8
    Float32    = 4
    Float64    = 8
    Function   = Uintptr
    Int        = Uint
    Int16      = 2
    Int32      = 4
    Int64      = 8
    Int8       = 1
    Interface  = Uintptr << 1
    Map        = Uintptr
    Rune       = Int32
    Slice      = Uintptr + Int<<1
    String     = Uintptr + Int
    Uint       = word
    Uint16     = 2
    Uint32     = 4
    Uint64     = 8
    Uint8      = 1
    Uintptr    = Uint
    word       = 4 << (^uint(0) >> 32 & 1) // 4 or 8
)

We would also want to fixup math/bits and strconv:

package bits

import "sizeof"

// UintSize is the width of a uint in the current runtime.
const UintSize = sizeof.Uint << 3

package strconv

import "math/bits"

// IntSize is the size in bits of an int or uint value.
//
// Deprecated: please use math/bits.UintSize.
const IntSize = bits.UintSize

[CAFxX]

Interface, Slice and String have different sizes on 32 bit architectures.

[carnott-snap]

Good catch, I have redefined them based on their implementations in runtime:

• Slice: runtime.slice (actually a struct, thus a constant size value type of a pointer and two ints)
• Interface: runtime.iface (actually a struct containing two pointers)
• Map: runtime.hmap (always a pointer to a struct, thus a pointer)
• String: runtime.stringStruct (as the name implies, a struct containing a pointer and int)
• Channel: runtime.hchan (always a pointer to a struct, thus a pointer)
• Function: runtime.funcval (always a pointer to a struct, thus a pointer)

[gopherbot]

Change https://golang.org/cl/241679 mentions this issue: sizeof: new package with constants for Int, Uint, Uintptr, Int8, etc.

[gopherbot]

Change https://golang.org/cl/242018 mentions this issue: sizeof: consolodate hardware size constants

[mdempsky]

Was it considered to add a new builtin function sizeof that can accept both type and expression arguments (like C's sizeof operator) and returns untyped constants (so it's usable where unsafe.Sizeof isn't)? E.g., sizeof(int) seems as friendly as sizeof.Int, and it generalizes easily to non-primitive types (e.g., #40169).

[gopherbot]

Change https://golang.org/cl/242783 mentions this issue: cmd/compile: add builtin "sizeof" function

[mdempsky]

CL 242783 is a proof-of-concept for adding a new sizeof builtin variant of unsafe.Sizeof that returns untyped constants, and extends unsafe.Alignof and unsafe.Sizeof to accept type expressions as arguments in addition to value expressions.

[mdempsky]

Notably, I proposed extending unsafe.Alignof and unsafe.Sizeof to accept type arguments back in 2015: https://github.com/golang/go/issues/12994

It was rejected because "I think this simplifies very few real programs." Where's the data that sizeof.Int simplifies more than a few real programs?

The regular expression I posted in #12994 now lists 166 instances of unsafe.Alignof/unsafe.Sizeof within the standard library (up from 105 in 2015). It finds 83 instances in third-party Go libraries used internally within Google (file:third_party/golang).

By comparison, CL 242018 finds just 6 instances in the standard library that benefit from this (which could have been refactored using an "internal/sizeof" package already). The same internal Google codebase has just 12 instances of ^uint(0) >> 63 (as found by regular expression: uint\(0\)\ *>>\ *63). Of those 12, only 1 actually "needs" to be untyped (it's exported as an untyped constant, but from an internal package, and all uses already explicitly convert it to int), and only 1 would require adding an explicit conversion from uintptr to int to pass to strconv.ParseInt if replaced with unsafe.Sizeof. The rest are all either explicitly converted anyway, or only used in the RHS of a shift operation. (1 was completely unused, and another appeared in a comment.)

Edit: The internal Google code base also has 3 instances of ^uintptr(0)>>63, but all of these could use unsafe.Sizeof(uintptr(0)) instead. None of them need to be untyped.

[rsc]

@mdempsky, commenting on an accepted issue to suggest new ideas isn't something we typically see.
It would be better to file a new proposal suggesting exactly what you want to propose.
I don't see a description of what you're proposing, only a CL.
After you file it, please move it into the Proposal project Incoming.
Thanks.

[rsc]

FWIW, I worry about sizeof(x) overfitting to expectations from C.

I spoke to Rob about this, since he suggested the package approach in the first place, and he pointed out that having a package means we can extend it in interesting ways in the future, including things like calculating the actual memory size of a linked data structure, or a map, or a slice, or some other thing entirely, none of which is possible if we instead put C's sizeof(x) into the Go spec.

[rsc]

Given the other suggestions that have come in related to this package, it seems like we should move it back into active discussion. In the previous comment I mentioned that having a package means we can have more interesting functionality like sizeof.Map - how much memory does the entire map take up? - and so on. So maybe we should put off introducing these basic constants until we have a full picture of what we want the package to be.

I've put it on hold, though, since we don't have bandwidth to elaborate the other features right now.

[go101]

By adding SizeofInt, doesn't it mean the arbitrary precision int proposal is given up?

[bcmills]

@go101, I don't think that proposal is really viable for the type literally named int anyway.
(It doesn't fit the model of http://golang.org/design/28221-go2-transitions#language-redefinitions.)

I think if that proposal were adopted, it would have to introduce a new type with a name other than int, in which case sizeof.Int would not be misleading anyway.