0

Reading the mathematical definition of convexity (https://en.wikipedia.org/wiki/Convex_function), it seems that there is a relatively clear definition as to what makes a function "convex":

enter image description here

However, the above definitions of convexity seem to only apply to deterministic functions - my question relates to determining the convexity of functions involving random variables (i.e. non-deterministic functions).

For example, in the context of machine learning and neural networks (a type of machine learning algorithm), we are often interested in optimizing a "loss function" for the purpose of creating a model that represents our observed data as closely as possible. For neural networks, the "loss function" involves a "random variable" and can be represented in the following form:

enter image description here

My Question: Have any theoretical results been established regarding convexity definitions for non-deterministic functions (e.g. functions with random variables)? Or can some "basic math" be used to show that non-deterministic functions automatically fail to satisfy either of the above definitions of convexity for deterministic functions - and therefore, non-deterministic functions (also called "noisy functions") are automatically non-convex?

Can someone please comment on this?

Thanks!

References:

stats_noob
  • 4,375
  • If I understand correctly, then a "non-deterministic function" in your sense is a function on a measure space. There is no general definition of convexity. – WhatsUp Jan 17 '22 at 06:53
  • @ WhatsUp: thank you for your reply! This is interesting! If in such cases, there does not even exist a definition for convexity to begin with - it is unlikely that the desirable properties (convergence) of optimization algorithms like gradient descent that require the function being optimized to be convex .... will not be applicable. I.e. gradient descent is not guaranteed to converge on functions that are non-convex, and in my case, there is no general definition of convexity. Thanks! – stats_noob Jan 17 '22 at 07:00
  • 1
    @stats555 please use MathJax, pick one of the loss functions from that table and edit your question so that we understand its definition completely. We don't necessarily want to read the entire paper if the question is only if the function is convex. Reason I ask: I don't see what that random variable should be exactly. Secondly, all functions I see in the table look convex to me. – Kurt G. Jan 17 '22 at 11:57
  • Neutal networks are not a "machine learning algorithm" but a model. – Dirk Jan 22 '22 at 07:37

1 Answers1

2

I think what you're missing here is that the losses optimized in deep learning are (at least in principle) expectations, not the functions of the random variables themselves. We aim to minimize the expectation and, since this is generally too cumbersome, we settle for minimizing the empirical risk which is a finite-sum over some observed set (usually the training set).

Jürgen Sukumaran
  • 7,901
  • Agreed! One question: Is the minimization of the risk just cumbersome or rather impossible? (I'd say the latter, because one does not know the prob distribution behind the expectation.) – Dirk Jan 22 '22 at 07:38
  • Depends on the model/loss but most often it's impossible like you said. – Jürgen Sukumaran Feb 01 '22 at 11:17
  • ? If you don't know the distribution you have no chance to minimize its expectation of any loss (well, trivial losses the $\equiv const$ work…). – Dirk Feb 02 '22 at 07:19
  • What I meant was that for some problems you do know the distribution and can compute the expected values. They are effectively "toy" problems but they exist. – Jürgen Sukumaran Feb 02 '22 at 19:54