It is interesting to think about the seeming contradiction here. Looking at the von neuman theorem you linked earlier, the specific theorem is about a rational agent choosing between several different options, and saying that if their preferences follow the axioms (no dutch-booking etc), you can build a utility function to describe those preferences. 

First of all, humans are not rational, and can be dutch-booked. But even if they were much more rational in their decision making, I don't think the average person would suddenly switch into "tile the universe to fulfill a mathematical equation" mode (with the possible exception of some people in EA). 

Perhaps the problem is that the utility function describing an entities preferences doesn't need to be constant. Perhaps today I choose to buy pepsi over coke because it's cheaper, but next week I see a good ad for coke and decide to pay the extra money for the good associations it brings. I don't think the theorem says anything about that, it seems like the utility just describes my current preferences, and says nothing about how my preferences change over time. 

"Humans do have a utility function"? I would say that depends on what one means by "have".

Does it mean that the value of a humans' life can in principle be measured, only that measure might not be known to the human? Then I would not be convinced – what would the evidence for this claim be?

Or does it mean that humans are imperfect maximizers of some imperfectly encoded state-action-valuation function that is somehow internally stored in their brains and might have been inherited and/or learned? Then I would also not be conviced as long as one cannot point to evidence that such an evaluation function is actually encoded somewhere in the brain.

Or does it simply mean that the observable behavior of a human can be interpreted as (imperfecty) maximizing some utility function? This would be the classical "as if" argument that economists use to defend their modeling humans as rational agents despite all evidence from psychology.

When I said "actual utility" I meant that which we cannot properly formalize (human welfare and other values) and hence not teach (or otherwise "give" to) the agent, so no, the agent does not "have" (or otherwise know) this as their utility function in any relevant way.

In my use of the term "maximization", it refers to an act, process, or activity (as indicated by the ending "-ation") that actively seeks to find the maximum of some given function. First there is the function to be maximized, then comes the maximization, and finally one knows the maximum and where the maximum is (argmax).

On the other hand, one might object the following: if we are given a deterministic program P that takes input x and returns output y=P(x), we can of course always construct a mathematical function f that takes a pair (x,y) and returns some number r=f(x,y) so that it turns out that for each possible y we have P(x)=argmax f(x,y). A trivial choice for such a function is f(x,y)=1 if y=P(x) and f(x,y)=0 otherwise. Notice, however, that here the program P is given first, and then we construct a specific function f for this equivalence to hold. 

In other words, any deterministic program P is functionally equivalent to another program P' that takes some input x, maximizes some function f(x,y), and returns the location y of that maximum. But being functionally equivalent to a maximizer is not the same as being a maximizer. 

In the learning agent context: If I give you a learned policy pi that takes a state s and returns an action a=pi(s) (or a distribution of actions), then you might well be able to construct a reward function g that takes a state-action pair (s,a) and returns a reward (or expected reward) r=g(s,a) so that when I then calculate the corresponding optimal state-action-quality-function Q* of this reward function, it turns out that for all states s, we have pi(s)=argmax Q*(s,a). This means that the policy pi is the same policy as the one that a learning process would have produced that searches for the policy that maximizes the long-term discounted sum of rewards according to reward function g. But it does not mean that the policy pi was actually determined by such a possible optimization procedure: the learning process that produced pi can very well be of a completely different kind than an optimization procedure.

That depends what you mean by "continuously improving until you reach a limit which is not necessarily the global limit".

I guess by "continuously" you probably do not mean "in continuous time" but rather "repeatedly in discrete time steps"? So you imagine a sequence r(s1) < r(s2) < ... ? Well, that could converge to anything larger than each of the r(sn). E.g., if r(sn) = 1 - 1/n, it will converge to 1. (It will of course never "reach" 1 since it will always below 1.) This is completely independent of what the local or global maxima of r are. They can obviously be way larger. For example, if the function is r(s) = s and the sequence is sn = 1 - 1/n, then r(sn) converges to 1 but the maximum of r is infinity. So, as I said before, unless your sequence of improvements is part of an attempt to find a maximum (that is, part of an optimization process), there is no reason to expect that it will converge to some maximum.

Btw., this also shows that if you have two competing satisficers whose only goal is to outperform the other and who therefore repeatedly improve their reward to be larger than the other agents' current reward, this does not imply that their rewards will converge to some maximum reward. They can easily be programmed to avoid this by just outperforming the other by an amount of 2**(-n) in the n-th step, so that their rewards converge to the initial reward plus one, rather than to whatever maximum reward might be possible.