Various definitions of higher derivatives:

- recursive: $f^{(k)}=(f^{(k-1)})'$
  the $k$th derivative is the derivative of the $(k-1)$st derivative
NB: this requires -local- $C^{k-1}$ to define the $k$th derivative

- Taylor:
  The Taylor poly is the unique approximation to $o((x-a)^n)$;
  in particular, this is an -infintesimal- definition.
EG:
$$\begin{cases}x^3 & x\in \bQ\\ 0 & x \nin \bQ \end{cases}$$
is not even continuous for $x\neq 0$, but has $2$ derivatives at $0$,
in this sense.
Indeed, this is The Correct Definition,
and importantly -clarifies- 
(see 09/taylor/01 for more)

- Schwarz:
The philosophy here is that
$D = \lim_{h\to 0} \frac{1}{h} \Delta_h$
(the derivative is the limit of scaled differences)
so $D^2$ should be $\lim_{h\to 0} \Delta_h \Delta_h$
[yes, we've swapped around limits; this is a significant difference:
 if you don't swap limits, you get the traditional ``must be
 differentiable in a nbhd to be twice diff''
 -- and in that case $D^2=D\circ D$;
 if you do swap limits, you get the nicer Taylor notion,
 though then $D^2 \neq D\circ D$]

Writing it out yields
$$f''(a) := \lim_{h\to 0} \frac{f(a+2h)-2f(a+h)+f(a)}{h}$$

[Yes, this gives a concrete formulation of Taylor derivatives,
 though it's not terribly enlightening unless you already understand
 what's going on.]
It should be obvious that the higher Schwarz derivatives have formulae
involving binomial coefficients: $1 -3 +3 -1$, etc.

Alternatively,
$\Id$ is convolution with the delta function $\delta$,
and $D^k$ is convolution with $(-1)^k \delta^{(k)}$;
you can take a limit of smooth (real analytic!) functions converging
  to $(-1)^k \delta^{(k)}$
such that convolving with them is an approximation of $\frac{1}{h} \Delta^k_h$,
if you like, and this is suggestive.
[Standard protocol is to take the Gaussian bell curve with decreasing
 variance to converge to $\delta$, then derivatives for the rest]

-------------------------------
Incidentally, the difference between ``differentiable on one side''
and ``differentiable on both'' is reflected in:
- taking the limit over $h>0$ (or $h<0$) or over all $h$
- using forward or backwards differences

So EG you could use
$$\lim_{h\to 0} \frac{f(a+h/2)-f(a-h/2)}{h}$$
[this corresponds to using the -average- of the forward and backwards
 difference -- and this is the characterization that corresponds
 to the $-\delta'$, above -- so long as you use a general enough integral]
which has the amusing property of not requiring $f$ to be continuous
at $x$!!
[but if it blows up, it has to blow up symmetrically]

Similarly, mixing forward and backwards differences (or just squaring
that ``average difference'') gives the following for the Schwarz
derivative:
$$f''(a) := \lim_{h\to 0} \frac{f(a+h)-2f(a)+f(a-h)}{h}$$
