A good way to conceptualize what’s going on is not the idea that balls become "spiky" in high dimensions – like the article says, balls are always perfectly symmetrical by definition. But it’s the box becoming spiky, "caltrop-shaped", its vertices reaching farther and farther out from the origin as the square root of dimension, while the centers of its sides remain at exactly +-1. And the 2^N surrounding balls are also getting farther from the origin, while their radius remains 1/2. Now it should be quite easy to imagine how the center ball gets more and more room until it grows out of the spiky box.
ColinWright 42 days ago [-]
But there is another way to think of the high-dimensional balls where "spikey" is the right visualisation.
Consider the volume of a cap. Take a plane that is 90% of the distance from the centre to the edge, and look at what percentage of the volume is "outside" that plane. When the dimension is high, that volume is negligible.
And when the dimension is really high you can get quite close to the centre, and still the volume you cut off is very small. In our 3D world the closest thing that has this property is a spike. You can cut off quite close to the centre, and the volume excised is small.
The sense in which a high-dimensional ball is not a spikey thing is in the symmetry, and the smoothness.
So when you want to develop an intuition for a high dimensional ball you need to think of it as simultaneously symmetrical, smooth, and spikey.
Then think of another five impossible things, and you can have breakfast.
dawnofdusk 42 days ago [-]
I think our geometric intuition could never be good for a high dimensional object. Consider a sphere in 3D. It's represented by the points which satisfy x^2 + y^2 + z^2 = 1. Because there are only three coordinates, knowing the value of one of them greatly reduces the possibilities for the other two. For example, if I know z is close to 1, then my point is close to the north pole (x and y are both close to zero).
However, if I have the n-sphere x_1^2 + x_2^2 + ... + x_n^2 = 1, knowing the value of x_1 gives me very little information about all the other coordinates. And humans' interaction with the geometry is reality is somewhat limited to manipulating one coordinate at a time, i.e., our intuition is for built on things like moving our body linearly through space, not dilating the volume or surface area of our bodies.
saagarjha 42 days ago [-]
If x_1 is close to one, then the more x_n values there are the closer they are to zero.
Hugsun 42 days ago [-]
That's a good point. High dimensional objects can obtain very unintuitive properties, like you describe.
This to me feels similar in many ways to how a corner in a high dimensional n-cube, although 90 degrees, no matter how you measure it, seems extremely spiky. As the shape does not increase in width, but the corners extend arbitrarily far away from the center. A property reserved for spiky things in 3D.
pfortuny 43 days ago [-]
Exactly: a corner of a square covers 1/4 of that part of the plane. A corner of a cube covers 1/8, a corner of a hypercube in dimension n covers just 1/(2^n) of the space. But each side/face/hyperface divides the plane/space/n-dim space just in half.
beretguy 42 days ago [-]
Ok, now i started to understand something. Thank you.
pfortuny 42 days ago [-]
In some sense, a ball is more “natural” than a cube in Euclidean n-space. Once you add the metric, cubes become artificial constructs (despite being the natural elements in just product spaces).
blaufuchs 42 days ago [-]
TFA does say this too, right after the Hamming lecture.
"So instead of considering n-balls to be spiky, it’s the space around them that outgrows them."
Sharlin 42 days ago [-]
Yes, but that doesn’t yet give me much intuition, so I wanted to elaborate on an idea I’ve found somewhat useful.
Hugsun 42 days ago [-]
To be clear: the commenter you replied to just seems to be reiterating the idea, so I'm not accusing them of not reading the article.
You should have seen how few replies read the last article I posted.
https://news.ycombinator.com/item?id=40525629
The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.
Sharlin 42 days ago [-]
"Space is growing around the ball" just by itself doesn’t give much of an intuition at least to me, so I just wanted to elaborate on a way to think about it.
Hugsun 42 days ago [-]
Yep, thanks for the great contribution!
ColinWright 42 days ago [-]
> The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.
I have thoughts about this, but I can find no way to contact you to open a conversation. You might want to think about adding something to your HN profile.
Meanwhile ... from this and other sources, I feel your pain.
gcanyon 42 days ago [-]
I read the article and the threads (to a point). It's hilarious that in the article you describe LLMs as being like hasty people jumping to conclusions, and the commenters on HN do exactly that thing in their comments.
Hugsun 31 days ago [-]
Haha, that's a really funny similarity. I appreciate you sharing it :)
petters 42 days ago [-]
Very good to see this as the top-voted comment. I completely agree that this seems like a more natural explanation of what is going on.
jheriko 43 days ago [-]
even then the edges do not suddenly curve. its just all round a bad analogy.
steventhedev 43 days ago [-]
This is a really good demonstration of the curse of dimensionality[0]
Interesting how this relates to LLMs scaling laws.
drdeca 43 days ago [-]
Why did I imagine that this would be about two shapes that are merely topologically n-balls, each having part of their boundary be incident with one of the two hemi(n-1)-spheres of the boundary of an n-ball (and otherwise not intersecting it)? (So like, in 3D, if you took some ball and two lumps of clay of different colors, and smooshed each piece of clay over half of the surface of the ball, with each of the two lumps of clay remaining topologically a 3-ball.)
I don’t know that there would even be anything interesting to say about that.
Hugsun 43 days ago [-]
I can't tell you why you imagined that, but that's pretty funny nevertheless.
robwwilliams 43 days ago [-]
Impressive, helpful, and now time to rebuild my own embeddings so I can grasp that red n-ball with my new n-D hands.
ColinWright 42 days ago [-]
For other HN discussions of this phenomenon you can see some previous submissions of another article on it.
That article doesn't have the nice animations, but it is from 14 years ago ...
It great to see these interesting math facts continue to be discussed and presented in new ways.
bt1a 43 days ago [-]
I am struggling to juggle the balls in my mind. Are there any stepping-stone visual pieces like this to hopefully get me there? Very neat write-up, but I can't wait to share the realized absurdity of the red ball's green box eclipsing in our 3D intersection of the fully diagonalized 10D construct
pfortuny 43 days ago [-]
The hypercube is the strange thing, not the red sphere. Placing the blue spheres tangent to the hypercube is an artificial construct which only “bounds” the red sphere in small dimensions. Our intuition is wrong because we think of the problem the wrong way (“the red sphere must be bounded by the box”, but there is no geometrical argument for that in n dimensions).
badmintonbaseba 42 days ago [-]
How does the enclosed sphere's radius changes with the number of dimensions, if the enclosing spheres are the following:
2D: 3 mutually touching 2-spheres (circles)
3D: 4 mutually touching 3-spheres (or spheres)
...
This variation of the problem doesn't rely on an artificial construct of a hypercube, I wonder if this yields a similarly unintuitive result.
badmintonbaseba 42 days ago [-]
If my calculations are correct, then for this variation the enclosed n-sphere's radius converges to sqrt(2)-1 from below, and remains enclosed in the bounding hyper-tetrahedron.
Hugsun 29 days ago [-]
Very interesting, I've considered doing something similar with other regular polyhedra, like the n-simplex (the one you analyzed) and n-orthoplex.
What was the side length in your calculation? did you find an equation for the size of the center n-ball?
pfortuny 42 days ago [-]
Buf, you may be right but I just cannot visualize it. It took me quite a while to do for the cube, imagine a tetrahedron. But you might be right.
Imustaskforhelp 43 days ago [-]
Can I just say how my mind is utterly blown by the animations
Hugsun 43 days ago [-]
Thank you <3
The trigonometry involved was pretty intense at times.
wow discovering Hamming’s lecture was enough for me! so good
Scarblac 42 days ago [-]
The whole series is on Youtube, awesome.
The course was also the basis for his book _The Art of Doing Science and Engineering_ (1997). At first it takes some getting used to as you have the feeling it may be outdated, but it's about teaching a style of thinking. It's great.
blaufuchs 42 days ago [-]
I know, I didn't realize he was alive in the 90s! Hearing him (sarcastically) say "now having 10 parameters isn't unusual, correct?" makes me wish he could've seen the 60B-parameter curve fitting we're doing nowadays.
Scarblac 42 days ago [-]
The whole lecture series was about giving students a style of thinking that might hopefully prepare them for the future, without focusing on special knowledge (there were other courses for that).
Two of the lectures were spent on building intuition for very high dimensionality (this one), and another on neural networks, because he thought there was a big chance they were going to be important. In the early 90s, not bad.
f1shy 42 days ago [-]
I think he already knew something about it... it talks about AI. Maybe at the time 100k to 1M dimensions? A bright mind like his, could very good extrapolate to 2024.
f1shy 42 days ago [-]
Yes sir! I've just bookmarked the series... Priceless!
WhitneyLand 43 days ago [-]
Both ChatGpt 4.o and Claude failed to answer
“…At what dimension would the red ball extend outside the box?”
If anyone has o1-preview it’d be interesting to hear how well it does on this.
V__ 43 days ago [-]
This was the prompt I gave o1-preview:
> There is a geometric thought experiment that is often used to demonstrate the counterintuitive shape of high-dimensional phenomena. We start with a 4×4 square. There are four blue circles, with a radius of one, packed into the box. One in each corner. At the center of the box is a red circle. The red circle is as large as it can be, without overlapping the blue circles. When extending the construct to 3D, many things happen. All the circles are now spheres, the red sphere is larger while the blue spheres aren’t, and there are eight spheres while there were only four circles.
> There are more than one way to extend the construct into higher dimensions, so to make it more rigorous, we will define it like so: An n-dimensional version of the construct consists of an n-cube with a side length of 4. On the midpoint between each vertex and the center of the n-cube, there is an n-ball with a radius of one. In the center of the n-cube there is the largest n-ball that does not intersect any other n-ball.
> At what dimension would the red ball extend outside the box?
Response: "[...] Conclusion: The red ball extends outside the cube when n≥10n≥10."
It calculated it with a step-by-step explanation. This is the first time I'm actually pretty stunned. It analysed the problem, created an outline. Pretty crazy.
Hugsun 43 days ago [-]
I'd wager that it's in the training data.
43 days ago [-]
Asraelite 42 days ago [-]
I never understood the need to distinguish between "ball" and "sphere" in maths. Sure, one is solid and the other hollow, but why is that fact so important that you need to use a completely different word? As I understand it, you could replace every instance of "ball" in this article with "sphere" and it would still be correct.
We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?
Hugsun 42 days ago [-]
> you could replace every instance of "ball" in this article with "sphere" and it would still be correct.
I completely agree. N-ball is maybe more mathematically precise. It might rule out some strange edge case I couldn't think of. I chose it mostly for stylistic reasons. n-ball is shorter than n-sphere, and ball is a more playful term.
I do however understand the need to define the two object classes, and see little issue with giving them distinct names.
42 days ago [-]
bmacho 42 days ago [-]
> We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?
Topologists don't need them because they already have ball and sphere.
In analysis, I can imagine them calling full hyperrectangles "brick", and empty hyperrectangles "box", but both words start with "b", so there is no shorthand for them on paper. I^n and ∂I^n are just fine.
NeoTar 42 days ago [-]
And, indeed, it’s probably in topology where the distinction between sphere and ball is most important.
NeoTar 42 days ago [-]
We do for 2D though - Circle and disc. And in 4D plus we have the 3-sphere and 4-ball, etc.
And for 3D shapes have Torus, and, well, ‘solid-torus’.
Consider the volume of a cap. Take a plane that is 90% of the distance from the centre to the edge, and look at what percentage of the volume is "outside" that plane. When the dimension is high, that volume is negligible.
And when the dimension is really high you can get quite close to the centre, and still the volume you cut off is very small. In our 3D world the closest thing that has this property is a spike. You can cut off quite close to the centre, and the volume excised is small.
The sense in which a high-dimensional ball is not a spikey thing is in the symmetry, and the smoothness.
So when you want to develop an intuition for a high dimensional ball you need to think of it as simultaneously symmetrical, smooth, and spikey.
Then think of another five impossible things, and you can have breakfast.
However, if I have the n-sphere x_1^2 + x_2^2 + ... + x_n^2 = 1, knowing the value of x_1 gives me very little information about all the other coordinates. And humans' interaction with the geometry is reality is somewhat limited to manipulating one coordinate at a time, i.e., our intuition is for built on things like moving our body linearly through space, not dilating the volume or surface area of our bodies.
This to me feels similar in many ways to how a corner in a high dimensional n-cube, although 90 degrees, no matter how you measure it, seems extremely spiky. As the shape does not increase in width, but the corners extend arbitrarily far away from the center. A property reserved for spiky things in 3D.
"So instead of considering n-balls to be spiky, it’s the space around them that outgrows them."
You should have seen how few replies read the last article I posted. https://news.ycombinator.com/item?id=40525629 The majority of the comments, including all the top ones, expressed insights as original, that were pretty thoroughly analyzed in the article. Just read my mildly frustrated replies.
I have thoughts about this, but I can find no way to contact you to open a conversation. You might want to think about adding something to your HN profile.
Meanwhile ... from this and other sources, I feel your pain.
[0]: https://en.m.wikipedia.org/wiki/Curse_of_dimensionality
I don’t know that there would even be anything interesting to say about that.
That article doesn't have the nice animations, but it is from 14 years ago ...
https://news.ycombinator.com/item?id=12998899
https://news.ycombinator.com/item?id=3995615
And from October 29, 2010:
https://news.ycombinator.com/item?id=1846682
It great to see these interesting math facts continue to be discussed and presented in new ways.
2D: 3 mutually touching 2-spheres (circles)
3D: 4 mutually touching 3-spheres (or spheres)
...
This variation of the problem doesn't rely on an artificial construct of a hypercube, I wonder if this yields a similarly unintuitive result.
What was the side length in your calculation? did you find an equation for the size of the center n-ball?
The course was also the basis for his book _The Art of Doing Science and Engineering_ (1997). At first it takes some getting used to as you have the feeling it may be outdated, but it's about teaching a style of thinking. It's great.
Two of the lectures were spent on building intuition for very high dimensionality (this one), and another on neural networks, because he thought there was a big chance they were going to be important. In the early 90s, not bad.
“…At what dimension would the red ball extend outside the box?”
If anyone has o1-preview it’d be interesting to hear how well it does on this.
> There is a geometric thought experiment that is often used to demonstrate the counterintuitive shape of high-dimensional phenomena. We start with a 4×4 square. There are four blue circles, with a radius of one, packed into the box. One in each corner. At the center of the box is a red circle. The red circle is as large as it can be, without overlapping the blue circles. When extending the construct to 3D, many things happen. All the circles are now spheres, the red sphere is larger while the blue spheres aren’t, and there are eight spheres while there were only four circles.
> There are more than one way to extend the construct into higher dimensions, so to make it more rigorous, we will define it like so: An n-dimensional version of the construct consists of an n-cube with a side length of 4. On the midpoint between each vertex and the center of the n-cube, there is an n-ball with a radius of one. In the center of the n-cube there is the largest n-ball that does not intersect any other n-ball.
> At what dimension would the red ball extend outside the box?
Response: "[...] Conclusion: The red ball extends outside the cube when n≥10n≥10."
It calculated it with a step-by-step explanation. This is the first time I'm actually pretty stunned. It analysed the problem, created an outline. Pretty crazy.
We don't have special words for the voluminous versions of other 3D shapes, so why do spheres need one?
I completely agree. N-ball is maybe more mathematically precise. It might rule out some strange edge case I couldn't think of. I chose it mostly for stylistic reasons. n-ball is shorter than n-sphere, and ball is a more playful term.
I do however understand the need to define the two object classes, and see little issue with giving them distinct names.
Topologists don't need them because they already have ball and sphere.
In analysis, I can imagine them calling full hyperrectangles "brick", and empty hyperrectangles "box", but both words start with "b", so there is no shorthand for them on paper. I^n and ∂I^n are just fine.
And for 3D shapes have Torus, and, well, ‘solid-torus’.
https://en.m.wikipedia.org/wiki/MacGyver
Reboot on TV this year.