I took a similar route as you, worked out the math and came to the realization that the math would result in the first character being one of 4 possibilities. At this point, I realized that using multiply or divide to perform math as a substitute for shift left and shift right was causing inaccuracy and wasn't ideal for decoding. I then converted a few characters to binary and manually performed the shifts and other calcs on them, and it became quickly apparent that what was happening was it was doing something similar to base64 where 3 8-bit characters were being combined to 24 bits and then being split into 4x 6-bit groupings.

I only mention this because it was a good lesson for me to learn for future CTFs. It is not always safe to use math to substitute shifts :) Once I realized the shift, I was able to quickly convert it in notepad without any coding.