How to Use Proxies with Python for Optimal Results

Share this article

How to use Proxies with Phyton

Python is perhaps one of the strongest programming languages, and it can be used basically everywhere, from web development to Artificial Intelligence.In this guide we will show you, how you can use proxies with Python. Python can help create some powerful applications. One another region where Python is extensively used occurs well in web scraping.

Web Scraping is the method through which you can extract data from your favorite website and then use that data according to your needs. However, one of the challenges that you face while doing web scraping is rate limits, which is simply to say if you want to scrape a large volume of data, then you have to make HTTP requests to your target website frequently now this can lead to blocking as websites mark this as suspicious activity.

One way around this limitation is by using proxies. Proxies are well like intermediaries between your device and the internet. They act like a gateway. They can help you access some geo-restricted content and can also help you with web scraping, as they can make your HTTP request appear to come from different devices.

Understanding Proxies

http vs https vs Socks for using proxies with python

At its core, a proxy is like a server that receives your request and then forwards it to your target’s website. Your target website then, in return, sends the response back to your proxy, which is then handed to you. This way, your real IP address is kept anonymous, and once provided, you with the proxy is used.

There are different types of proxies available that you can use; here are some of the basic ones:

HTTP Proxies

By sending HTTP requests and replies, HTTP proxies make it easier for users and web servers to talk to each other. By saving material that is often viewed, they can improve network speed, protect privacy, and get around geo-blocks. However, HTTP proxies might not secure data, which could be bad for security, and they usually can't handle information that isn't HTTP.

HTTPS Proxies

These types of proxies are capable of handling both HTTP and HTTPS requests, providing a secure channel for data transmission through encryption via SSL/TLS protocols. While HTTPS proxies offer increased security compared to HTTP proxies, the encryption process may lead to a slight decrease in speed. Despite this, they are preferred for their added layer of protection, especially when dealing with sensitive information online.

SOCKS Proxies

SOCKS proxies stand out from other types due to their ability to handle various types of traffic, including HTTP and HTTPS, while routing data through multiple intermediary servers, enhancing anonymity. However, one drawback associated with SOCKS proxies is their relatively slower performance compared to other proxy types mentioned earlier. Additionally, their setup and configuration tend to be more complex, requiring users to invest more time and effort in implementation and maintenance.

Choosing a Proxy Provider

There are many proxy providers out there, offering all sorts of different proxies with different locations and different features at different prices.

Before picking any one of them, there are a few things that you should keep in mind; some of them are:

Choose the Type of your Proxy

The type of proxy is one of the most important factors you should keep in mind while purchasing a proxy service; depending on your needs, you might want to get the proxy. There is a wide list of proxy types; search them out first.

Choose the Location of your Proxy

Accessing geo-restricted content requires the IP address of a device residing in the country where that website is accessible, so you must choose the proxy provider supporting that particular country.

Choose a Provider with High Quality Services

We all love it when our tasks run smoothly, right? Unfortunately, not all proxies work smoothly; going with something cheap will make the experience less smooth for you. Similarly, a high-quality proxy means a relatively better uptime.

Keeping all the factors in mind, it’s safe to say that if you want all the factors mentioned above to be green-ticked, you should try PetaProxy. They offer both HTTP and SOCKS services. PetaProxy also provides you with different types of proxies, such as Mobile proxies and Datacenter proxies.

Petaproxy Advantages

Now, after deciding on the proxy provider, the next thing is how to use this within your scraping. Here’s the step-by-step guide; make sure to follow each step sequentially.

Installing the Required Libraries

To use proxies, you will need two libraries, particularly requests and requests[socks]. Requests is a library that allows you to make HTTP requests to your target website in Python, whereas the other requests[socks] will enable you to use SOCKS proxies with requests

Before installing them, you must ensure you have Python installed on your device. In order to install them, you can use the pip (package manager for Python).

Run the following command in the cmd to install these libraries.

				
					pip install requests
pip install requests[socks]
				
			

Using Proxies with Python Requests

After installing both of the libraries, the next thing that we want to do is import both of the libraries; you can import them using the following statements.

				
					import requests 
				
			

You don’t need a separate import statement for the requests[sock] because the handling SOCKS proxies are automatically integrated into the requests library. Now, what you can do to use proxies with Python is pass a dictionary of the proxy setting to the proxy parameter of the request.get() method.

The dictionary should have a format that looks like this:

				
					proxies = {
    "http": "http://username:password@proxy_ip:proxy_port",
    "https": "https://username:password@proxy_ip:proxy_port",
    "socks5": "socks5://username:password@proxy_ip:proxy_port"
}
				
			

Now, you need to replace the username, password, proxy_ip, and proxy_port with your own proxy credentials. Some proxies don’t require authentication; if your proxy is one of them, you can also omit the username and password, and your code would look like this:

				
					proxies = {
    "http": "http://proxy_ip:proxy_port",
    "https": "https://proxy_ip:proxy_port",
    "socks5": "socks5://proxy_ip:proxy_port"
}
				
			

Note: Make sure to replace proxy_ip and proxy_port with your own proxy values.

Now, if you want to use different proxies for different protocols, you can certainly do that.

Here’s the way you can do that;

				
					# Different proxies for each protocol
proxies = {
    "http": "http://http_proxy_ip:http_proxy_port",
    "https": "https://https_proxy_ip:https_proxy_port",
    "socks5": "socks5://socks5_proxy_ip:socks5_proxy_port"
}
				
			

If you want to use the same proxy for all the protocols, your code should look like this;

				
					# Use the same proxy for all protocols
proxies = {
    "all": "http://proxy_ip:proxy_port"
}
				
			

Choose this according to your own needs. 

Now in order to make the request, you can use this code:

				
					import requests

# Replace the following values with your own proxy configuration
http_proxy_ip = "your_http_proxy_ip"
http_proxy_port = "your_http_proxy_port"
https_proxy_ip = "your_https_proxy_ip"
https_proxy_port = "your_https_proxy_port"
socks5_proxy_ip = "your_socks5_proxy_ip"
socks5_proxy_port = "your_socks5_proxy_port"

# Different proxies for each protocol
proxies = {
    "http": f"http://{http_proxy_ip}:{http_proxy_port}",
    "https": f"https://{https_proxy_ip}:{https_proxy_port}",
    "socks5": f"socks5://{socks5_proxy_ip}:{socks5_proxy_port}"
}

# Alternatively, if you want to use the same proxy for all protocols
# proxies = {"all": f"http://proxy_ip:proxy_port"}

# Replace the URL with the target URL you want to request
target_url = "https://www.example.com"

try:
    # Make a request with proxies
    response = requests.get(target_url, proxies=proxies)

    # Check if the request was successful (status code 200)
    if response.status_code == 200:
        print("Request successful!")
        print(response.text)
    else:
        print(f"Request failed with status code {response.status_code}")

except requests.RequestException as e:
    print(f"An error occurred: {e}")
				
			

Now, make sure to replace all of the variables according to your needs. This code creates a GET request to the target website and then prints out the response text we get from the request.

In a similar manner you also make use of requests.post(), so that you can also make a POST in a similar manner.

Compared to the GET method (which appends the data to the URL), POST requests send data in the request’s body. This is very useful when sending in a large amount of data, such as login credentials or search queries.

				
					try:
    # Make a POST request with proxies
    response = requests.post(target_url, proxies=proxies, data=payload)

    # Check if the request was successful (status code 200)
    if response.status_code == 200:
    #prints out the following if it successfully sent the request

        print("POST request successful!")
        print(response.text)
    else:
        #prints out the following if it there is the post request is not successful
        print(f"POST request failed with status code {response.status_code}")

except requests.RequestException as e:
    print(f"An error occurred: {e}")
				
			

Now what code does, is make POST requests to the target website using the proxies that we defined. Then it will pass the data dictionary as the payload (which will be containing the query it searches out), it will then print out the status code and response text of the request.

Conclusion

To sum up, we learnt how we can actually make use of proxies in Python. Request is a popular library that you can use inorder to make HTTP requests in Python. Other than that we learnt about proxies, what they really are, and what factors you should keep in mind while deciding on a proxy provider, how you can install libraries using the pip command and then you can use get() and post() functions.

Philipp Neuberger

Managing Director

Philipp, a key writer at Petaproxy, specializes in internet security and proxy services. His background in computer science and network security informs his in-depth, yet accessible articles. Known for clarifying complex technical topics, Philipp’s work aids both IT professionals and casual internet users.

Don’t miss anything! Sign up for our newsletter

Your One-Stop Source for Trending Updates and Exclusive Deals

We respect your privacy. Your information is safe and you can easily unsubscribe at any time.

Table of Contents

BEST OFFER

Petaproxy Logo
Petaproxy Trustpilot
Petaproxy Logo
Petaproxy Trustpilot
Out of stock

Prepare for Launch this February!

February marks the arrival of our cutting-edge proxy service. Be among the first to elevate your internet freedom and security as we launch.

Early Bird Special: Sign up now for an exclusive 20% discount on our launch!