OK, I've done some sketches and I've figured out how I want things to work and the math(s) behind it. Now the hard bit is to bend the math so that it fits Vectrex hardware. I guess I should point out what makes things tricky.
1 No floating point numbers (no fractions or numbers between 0 and 1)
2 No trigonometric functions
3 No divide! (although you can write your own)
1 I use two bytes for my numbers, I guess you could call them the ones and 1/256s columns (as opposed to 1s and 0.1s). I have my own routines that deal with signing, adding and multiplying these numbers. Multiplying fractions it especially useful when we get to (2)
2 I use pre-calculated lookup tables (normally produced in MS Excel) these lookups map angles in Vegrees (256 vegrees in a circle) to my byte fractions in 1/256s
3 Simple integer division I do manually with subtraction. Non integer division I try to avoid or multiply by reciprocals.
Since I use lookups for my trig I need to decide which ones to use. Normally Sin and Cos are all I need, and these tend to be in 2 bytes form. SSSS used only a 1 byte Arctan table. I figure the new game will need Sin, Cos and Tan. So I add to my existing Sin and Cos tables using Excel spreadsheet exported as CSV.
I figure it's about time I powered up the Vectrex to start coding!