tl;dr
1. Figure out a way to generate numbers divisible by N
2. Generate a few numbers and find their GCD. It is likely equal to N
3. Find G from calculate(0) and calculate(1)
4. Find every bit of H by flipping the bit and looking at how flipping one bit in the input affects the output of function "calculate"

Original writeup (https://github.com/hnn4abo/ctf/tree/master/viettel-mates-ctf-2018/super-encryption-system).