Long story short: we used *frida* to patch the code and let it compute the solutions.
You can find all the details by following the provided link (https://zenhack.it/writeups/XmasCTF2018/catana/)