The simplest way to convert the square into an octagon is to divide each of the sides of the square into 3 segments of 3". The central segment of each side will form a 3" side of the octagon, and cutting across the corners will give you a side of length 3sqrt(2)=4.2426" approx. So the octagon is irregular with four sides equal to 3" and four sides equal to 4.2426". The lengths of the adjacent sides alternate between 3" and 4.2426".
For a regular octagon the centre of the square and the centre of the octagon coincide. The central angle of the octagon measures 360/8=45, so the octagon consists of 8 isosceles triangles where the vertex angle is 45 degrees. The height of each triangle is half the side of the square=4.5". To find the length of the side of the regular octagon, we consider one isosceles triangle of height 4.5" and we split it into two back-to-back right-angled triangles. The vertex angle is bisected so that we can write (a/2)/4.5=tan(22.5), where a is the side of the octagon. So a=9tan(22.5)=3.728" approx. [tan(45)=2tan(22.5)/(1-(tan(22.5)^2); tan45=1, so 1-tan(22.5)^2=2tan(22.5). Let x=tan(22.5), then x^2+2x=1, x^2+2x+1=2; (x+1)^2=2; x+1=sqrt(2); x=tan(22.5)=sqrt(2)-1; a=9(sqrt(2)-1).] So, a regular octagon will have a side length 3.728" approx. if the square has side 9".