How to use Overplay and other VPNs as a cURL proxy

UPDATE May 2015

When I posted this tutorial two years ago, I gradually got comments that the OpenVPN interface was setup successfully but curl traffic through the interface timed out. I tried to help but responded with the cliched ‘works on my machine’, with my machine being Debian Squeeze and kernel version 2.6. Recently I upgraded my server to Debian Jessie, kernel 3.14 and OpenVPN 2.3.4 and got the timeout problem that everyone was talking about. If I’d to guess, I’d say it’s some change in the kernel and routing tables but I really have no idea. But fear not, a random commentator from May 2014 called William Asssaad saved us all. We need to add a routing table for the VPN interface after it’s all setup. You can do it manually with the ip route add and ip rule add commands but I prefer to do it in a shell script, which I got from this blog post: http://snikt.net/blog/2013/10/10/how-to-force-program-to-use-vpn-tunnel/

Proxies are like hard drive space, you can never have enough. Or enough IPs to be more accurate as Facebook, Google and other services are getting better at flagging the IPs of popular HTTP/SOCKS5 proxies. So we need to find fresh proxies to use in our PHP/Python scripts. A great source is the proliferation of VPN services that are popping up as consumers worry more about their Internet privacy. Problem is these are intended for use by endusers on their desktops and not in serverside PHP scripts. So it’s a bit tricky to get these working with cURL, but fear not, I explain all in this post.

The magic of being able to use a VPN in cURL is the CURLOPT_INTERFACE option. This lets you set the network interface that cURL uses. You can’t use a VPN directly in cURL as cURL/PHP operates on a higher network level than the VPN protocol.

1)
So we need to setup the VPN on a new interface. Note that you absolutely need root access to your server to create interfaces so this guide is only useful for people with their own dedicated servers. People on shared $2/month servers are shit out of luck. You might get it working on a VPS, I’ve no idea. So to create an interface, you need to download and install OpenVPN if you haven’t it installed already. There’s loads of info online to help you do this, so figure it out and come back.

2)
Next we need to get the configuration files we need from our VPN provider of choice. I use Overplay.net. You need an account with them and it cost something like $5 a month. Download the ZIP file of connection files and unzip on your server in a new directory. Also download the Overplay public key certificate and make sure it’s in the same directory:
curl --insecure -o connections.zip https://patch.oppatch.com/connections.zip
unzip -u connections.zip
curl --insecure -o OverplayCert.crt https://patch.oppatch.com/OverplayCert.crt

Now we have the connection files which work fine if you run Linux as your desktop OS and just want to browse the web as described in this Overplay guide. But we DO NOT want to just start the VPNs as is as it will take over the main Internet connection and make your server inaccesible. I did this a few times and had to get my host to reboot my server.

3)
So we need to edit the configuration files and add one command, route-nopull. This prevents Overplay from taking over the routing information. If you take one thing away from this blogpost, it’s the addition of the route-nopull option as it’s what lets you use these config files on your server.

May 2015: we need to change route-nopull to these 3 commands I got from this blogpost: http://snikt.net/blog/2013/10/10/how-to-force-program-to-use-vpn-tunnel/
script-security 2
route-noexec
route-up /root/route_up.sh

route-up is the imporant bit, it runs the shell script at /root/route_up.sh when the interface is created. Here’s the contents of route_up.sh:

#!/bin/sh

echo "$dev : $ifconfig_local -> $ifconfig_remote gw: $route_vpn_gateway"

ip route add default via $route_vpn_gateway dev $dev table 20
ip rule add from $ifconfig_local table 20
ip rule add to $route_vpn_gateway table 20
ip route flush cache

exit 0

I also want to add my login as a file so I don’t have type it everytime. So create a new file in the same directory and name it ‘auth_overplay’ or whatever you want. Enter your username and password, seperated by a newline. So if we’re taking the ‘Overplay – Ireland-1.conf’ file as our example, our config would now look like this. Our additions are in bold at the bottom:
client
dev tun
proto udp
remote 176.67.85.4 1443

resolv-retry infinite
nobind
persist-key
persist-tun
ca OverplayCert.crt
verb 5
route-method exe
route-delay 2

tun-mtu 1500
tun-mtu-extra 32
mssfix 1450
comp-lzo

script-security 2
route-noexec
route-up /root/route_up.sh
auth-user-pass auth_overplay
#daemon


Note we also added daemon at the end, but commented it out. You can uncomment this when you’ve got everything working and want to start the VPN as a daemon so you can use it without having to have the SSH window open.

4)
Now start up the VPN with OpenVPN:
openvpn "Overplay - Ireland-1.conf"

If everything works, you should see output like this:
Fri Jul 26 21:05:13 2013 us=31236 OpenVPN 2.1.3 x86_64-pc-linux-gnu [SSL] [LZO2] [EPOLL] [PKCS11] [MH] [PF_INET6] [eurephia] built on Feb 21 2012
Fri Jul 26 21:05:13 2013 us=31325 WARNING: No server certificate verification method has been enabled. See http://openvpn.net/howto.html#mitm for more info.
Fri Jul 26 21:05:13 2013 us=31331 NOTE: OpenVPN 2.1 requires '--script-security 2' or higher to call user-defined scripts or executables
Fri Jul 26 21:05:13 2013 us=31678 LZO compression initialized
Fri Jul 26 21:05:13 2013 us=31725 Control Channel MTU parms [ L:1574 D:138 EF:38 EB:0 ET:0 EL:0 ]
Fri Jul 26 21:05:13 2013 us=31748 Socket Buffers: R=[124928->131072] S=[124928->131072]
Fri Jul 26 21:05:13 2013 us=31766 Data Channel MTU parms [ L:1574 D:1450 EF:42 EB:135 ET:32 EL:0 AF:3/1 ]
Fri Jul 26 21:05:13 2013 us=31777 Local Options String: 'V4,dev-type tun,link-mtu 1574,tun-mtu 1532,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-client'
Fri Jul 26 21:05:13 2013 us=31781 Expected Remote Options String: 'V4,dev-type tun,link-mtu 1574,tun-mtu 1532,proto UDPv4,comp-lzo,cipher BF-CBC,auth SHA1,keysize 128,key-method 2,tls-server'
Fri Jul 26 21:05:13 2013 us=31794 Local Options hash (VER=V4): 'd3a7571a'
Fri Jul 26 21:05:13 2013 us=31802 Expected Remote Options hash (VER=V4): '5b1533a2'
Fri Jul 26 21:05:13 2013 us=31811 UDPv4 link local: [undef]
Fri Jul 26 21:05:13 2013 us=31816 UDPv4 link remote: [AF_INET]176.67.85.4:1443
WRFri Jul 26 21:05:13 2013 us=37221 TLS: Initial packet from [AF_INET]176.67.85.4:1443, sid=a552afa0 928c908a
WFri Jul 26 21:05:13 2013 us=37266 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
WRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRWRFri Jul 26 21:05:13 2013 us=72079 VERIFY OK: depth=1, /C=UK/ST=LANCS/L=MANCHESTER/O=OVERPLAY.NET_LLP/OU=CA/CN=OVERPLAY_CA/emailAddress=ca@overplay.net
Fri Jul 26 21:05:13 2013 us=72206 VERIFY OK: depth=0, /C=US/ST=IL/L=Chicago/O=OVERPLAY.NET_LLP/OU=SERVERS/CN=vpn1-us/emailAddress=ca@overplay.net
WRWRWRWRWWRRWWWWRRRRWRWRFri Jul 26 21:05:13 2013 us=472472 Data Channel Encrypt: Cipher 'BF-CBC' initialized with 128 bit key
Fri Jul 26 21:05:13 2013 us=472487 Data Channel Encrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
Fri Jul 26 21:05:13 2013 us=472528 Data Channel Decrypt: Cipher 'BF-CBC' initialized with 128 bit key
Fri Jul 26 21:05:13 2013 us=472534 Data Channel Decrypt: Using 160 bit message hash 'SHA1' for HMAC authentication
WFri Jul 26 21:05:13 2013 us=472560 Control Channel: TLSv1, cipher TLSv1/SSLv3 DHE-RSA-AES256-SHA, 1024 bit RSA
Fri Jul 26 21:05:13 2013 us=472576 [vpn1-us] Peer Connection Initiated with [AF_INET]176.67.85.4:1443
Fri Jul 26 21:05:15 2013 us=598837 SENT CONTROL [vpn1-us]: 'PUSH_REQUEST' (status=1)
WRRWRWRFri Jul 26 21:05:15 2013 us=604235 PUSH: Received control message: 'PUSH_REPLY,redirect-gateway def1,dhcp-option DNS 176.67.84.100,route 10.204.0.1,topology net30,ping 10,ping-restart 120,ifconfig 10.204.0.54 10.204.0.53'
Fri Jul 26 21:05:15 2013 us=604257 Options error: option 'redirect-gateway' cannot be used in this context
Fri Jul 26 21:05:15 2013 us=604275 Options error: option 'route' cannot be used in this context
Fri Jul 26 21:05:15 2013 us=604293 OPTIONS IMPORT: timers and/or timeouts modified
Fri Jul 26 21:05:15 2013 us=604297 OPTIONS IMPORT: --ifconfig/up options modified
Fri Jul 26 21:05:15 2013 us=604301 OPTIONS IMPORT: --ip-win32 and/or --dhcp-option options modified
Fri Jul 26 21:05:15 2013 us=604472 TUN/TAP device tun0 opened
Fri Jul 26 21:05:15 2013 us=604485 TUN/TAP TX queue length set to 100
Fri Jul 26 21:05:15 2013 us=604507 /sbin/ifconfig tun0 10.204.0.54 pointopoint 10.204.0.53 mtu 1500
WFri Jul 26 21:05:17 2013 us=671302 Initialization Sequence Completed

Now open a new SSH session as root and enter ifconfig to see the list of network interfaces. You should see your proxy listed with the interface name tun0 or something like that.

Some OverPlay servers don’t complete the TLS handshake for me but I’m thinking that was because they were old IPs. Overplay seems to change actual proxy servers a lot.

5)
If you see the tun0 interface in ifconfig, then it worked, probably. Test it with cURL on the command line:
curl --interface tun0 http://icanhazip.com

Replace tun0 with whatever your interface is called. icanhazip.com is a simple site that outputs your IP and nothing else and has been online for 3 years so hopefully it stays online. If everything works, the site should output the IP of Overplays proxy and not your server IP.

To use the interface in your PHP scripts, you’d set it with something like this:
curl_setopt($curlh, CURLOPT_INTERFACE, "tun0");

I found from experience that PHP/cURL defaults to using the standard network interface if something doesn’t work, so it grabs pages using your real IP! This can be a disaster if you’re doing dodgy stuff and don’t want to get banned from FB or Google. For this reason, I always grab icanhazip.com in my scripts and check to make sure it’s not my server IP.

Well that’s about it. I also wrote a few PHP scripts that automatically download the Overplay config files, edit them, check what country the IP is and add them to a MySQL table. Then I’ve other PHP code that just queries the table grabbing a random proxy from the country I need. I’ll do a blogpost and share that code if anyone actually reads this one and comments. And also comment if you run into any trouble, I’m usually quick to answer.