I had kind of a similar problem which I'd managed to resolve.
First step:
From a web browser within your local LAN, type:
http:// or
https://mailservername/omaIf you get a "browser not compatible error" after logging in this would indicate that Activesync is functioning as expected.
That being the case, you can then create an A record, e.g.: oma.domain.com corresponding to the public IP address of your server, or if you prefer, the IP address of the ISA server (remember to create the rule passing SSL / 443 traffic to your exchange box)
You should also install a SSL certificate on your IIS server (can be self signed as long as you can bear the certificate error message when connecting)
Let me know if that helps, or if you need any further info.