The vertical cross-section of a cone is an isosceles triangle ABC, where the base is BC and A the vertex angle such that ∠ABC=∠ACB. The perpendicular bisector AN from A to BC is also the bisector of angle A. BN=CN=r the radius of the cone.

AN=h, the height of the cone whose volume is πr²h/3. The bisectors of angles B and C meet the bisector of angle A at O, which is the centre of the inscribed circle radius R=OP=OQ=ON. AB and AC are tangents to the circle at P and Q respectively. ∠OPA=∠OQB=90°.

There are 4 congruent triangles: OPB, ONB, OQC, ONC. BP=BN=CN=CQ=r. The inscribed circle is a vertical cross-section of the sphere inside the cone.

h=AN=AO+ON=AO+R, so AO=h-R, and AP²+OP²=AO². Therefore, AP²=(h-R)²-R²=h²-2Rh.

Also, AB=AP+BP=√(h²-2Rh)+r, and AB²=AN²+BN²=h²+r².

Therefore, (√(h²-2Rh)+r)²=h²+r²,

h²-2Rh+r²+2r√(h²-2Rh)=h²+r²,

r√(h²-2Rh)=Rh.

So, squaring both sides:

r²(h²-2Rh)=R²h², r²h-2Rr²=R²h, h(r²-R²)=2Rr², h=2Rr²/(r²-R²).

Since R is a given constant, we have h in terms of r, so we can express the volume of the cone in terms of r:

V=⅓πr²(2Rr²/(r²-R²))=⅔πRr⁴/(r²-R²).

The rate of change of volume with respect to r is:

dV/dr=(⅔πR)(4r³(r²-R²)-2r⁵)/(r²-R²)².

dV/dr=(⅔πR)(2r⁵-4r³R²)/(r²-R²)².

When the volume is maximum or minimum, dV/dr=0, so 2r⁵-4r³R²=0, and 2r³(r²-2R²)=0. r=0 is the minimum, and r=R√2 is the maximum.

When r=R√2, h=2Rr²/(r²-R²)=4R³/R²=4R and the maximum volume of the cone is ⅓π(2R²)(4R)=(8/3)πR³.

So for max volume, r=R√2 and h=4R.