Can somebody give me some more reasons why this should not bother me?
The fact that you are bothered means you are in good company. Historically, whenever the notion of "number" has had its meaning extended, there were people who were "bothered" by the extension and thought that "something just seems philosophically/intuitively wrong". Examples of this include:
the idea that zero is a number
the idea that numbers can be negative
the idea that numbers can be non-integer
the idea that numbers can be irrational (major crisis there!)
the idea that numbers can be non-real
the idea of infinity (and particularly the idea that there is more than one type of infinity).
What we can learn from this history is that, first of all, it is natural and common to feel discomfort when learning about new concepts that extend or generalize the meaning of concepts we were already familiar with (particularly related to the notion of "number"); and second of all, that that discomfort has not so much to do with the nature of the mathematical concepts that cause the discomfort, but more to do with the psychological attachment that humans tend to form with notions we learn and get used to. We simply tend to resist the idea that a word or concept that we have gotten used to interpreting in a certain way has now broadened its meaning.
If you got used to thinking of "infinity" as meaning one particular thing and find that the one-point compactification notion of infinity pushes you beyond that relatively narrow idea and causes you discomfort, think of that discomfort as being a tangible sign that your brain is expanding and rewiring itself. I find it a bit analogous to the pain (which many people find pleasurable, at least at moderate levels) that you get when you do physical exercise - people enjoy that feeling because they see it as a sign that their muscles are growing and that they're becoming more fit. In the analogous math context, the discomfort can be a small reminder that you are becoming smarter. (Conversely, if you are reading a math textbook and not feeling any discomfort, there's a decent chance you're not actually thinking hard enough about what you're reading to really digest the ideas being discussed.)
In general, when one learns higher mathematics, there is a point at which a student stops constantly asking "but why does this definition make sense" and learns to accept that the explanation of why a concept is interesting or useful often comes after (sometimes long after!) the definition has been formulated. In the interim period between learning the definition and getting the satisfaction of understanding why it's a good one, one often experiences a tension or mild discomfort. That's okay! And some people not only get used to this feeling, but even learn to savor it and enjoy the anticipation of the moment when everything will become clear.
Of course, in this particular case, you do not need to wait very long to understand why having only one infinity for different directions can be useful - some of the other answers discuss this and have given pretty good explanations. But my analysis above applies more generally to many situations where you will learn an abstract mathematical concept and it won't necessarily make sense right away or will appear to be at odds with some earlier knowledge you had. So I hope you find this point of view helpful.