Using proxies with Node-Fetch
Node-fetch is a helpful package that allows developers to use the same functionality found in web browsers for making HTTP requests, called window.fetch, in Node.js applications. It enables developers to fetch data from servers, such as JSON files or images, by sending asynchronous requests. Node-fetch provides features like sending GET and POST requests, setting headers, including credentials, and more. It is particularly useful for tasks like web scraping, where retrieving data from websites is required.
To get started, you need to meet the following requirements:
- Install Node.js: If you don’t have Node.js installed on your system, you can visit the downloads page of Nodejs.org and download the appropriate version for your operating system.
- Install node-fetch package: To install the node-fetch package, you can utilize the Node Package Manager (NPM) tool.
Please note that Node Fetch version 3 is an ESM-only module. For the purpose of this tutorial, we will be using version 2 to ensure compatibility with CommonJS.
- To proceed, open your terminal and navigate to the directory where you want to store your source code. Once you’re in the desired directory, execute the following command to install node-fetch:
npm install node-fetch@2
Node fetch code example
To demonstrate the usage of Node Fetch, let’s create a new file and save it as “check-ip.js”.
In the file, we need to load the node-fetch module. To do so, include the following line of code:
const fetch = require(“node-fetch”);
Now, you can use either the “then-catch” syntax to handle promises. Here’s an example of how to use it:
fetch('https://ip-api.com/')
.then(response => response.text())
.then(data => console.log(data))
.catch(error => console.error(error));
Alternatively, you can use the newer “try-catch” syntax to handle errors. Here’s an example of how to use it:
(async () => {
try {
const response = await fetch("https://ip-api.com/");
const data = await response.text();
console.log(data);
} catch (e) {
console.error(e.message);
}
})();
Save the file and open your terminal. To run the code, enter the following command in the terminal:
node check-ip.js
The output displayed in the terminal will be your IP address.
Integrating Proxies
To integrate proxies, we will start by installing the https-proxy-agent package.
Create a new file and save it as proxies.js. In this file, add the following lines of code to load the necessary packages:
const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');
Next, we need to create a variable to store the URL of the HTTP or HTTPS proxy.
Many proxy servers, including DataImpulse’s Residential proxy servers, require you to provide credentials. In such cases, you can construct the HTTP proxy URL with the appropriate user credentials using the following syntax:
const proxyUrl = `http://${YourProxyPlanUsername}:${YourProxyPlanPassword}@${proxyServer}`;
In the code snippet provided, we are utilizing three additional local variables to construct the proxy URL. These variables are used to store your username, password, and the proxy server address.
Residential Proxies
The example below demonstrates the construction of the https proxy or http proxy URL for DataImpulse residential proxies:
const proxyUrl = `http://<YourProxyPlanUsername>:<YourProxyPlanPassword>
@gw.dataimpulse.com:823`;
In the above example, replace USERNAME and PASSWORD with the credentials of your DataImpulse proxy plan.
Testing Proxies
Now, let’s proceed to test the proxies. This step is the same for Residential proxy.
To begin, create an instance of the HttpsProxyAgent class. The constructor of this class requires the proxy URL we have just created:
const proxyAgent = new HttpsProxyAgent(proxyUrl);
Lastly, we can pass the proxy agent as one of the optional parameters to the fetch method, specifically the agent parameter. This agent represents an HTTP(S) agent instance that we have created using the http-proxy-agent package.
Bringing everything together, the code for using Node fetch with proxies looks like this:
const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');
const proxyUrl = `http://<YourProxyPlanUsername>:<YourProxyPlanPassword>@gw.dataimpulse.com:823`;
(async () => {
try {
const proxyAgent = new HttpsProxyAgent(proxyUrl);
const response = await fetch("https://ip-api.com/", { agent:
proxyAgent });
const data = await response.text();
console.log(data);
} catch (e) {
console.error(e.message);
}
})();
You can execute this code to retrieve the IP address associated with the proxy.
To rotate proxies with Node Fetch, you can follow these steps:
- If you’re using DataImpulse Residential Proxies, proxy rotation is handled automatically, and you don’t need to perform external rotation.
- For DataImpulse Residential Proxies, the IP address can be randomly changed for each request or remain the same for up to 30 minutes.
By utilizing the automatic proxy rotation features of DataImpulse proxies, you can avoid IP bans from websites that restrict usage with the same proxy.
To select a proxy randomly from a proxy list and rotate them, you can use the following code:
const fetch = require('node-fetch');
const HttpsProxyAgent = require('https-proxy-agent');
proxyUrls =[
'127.0.0.1:10000',
'127.0.0.2:10001',
'127.0.0.3:10002'
]';
(async () => {
try {
for (let i = 0; i < 3; i++) {
const randomIndex = Math.floor(Math.random() * proxyUrls.length); // Generate a random index
const proxyAgent = new HttpsProxyAgent(proxyUrls[randomIndex]); // Select a proxy URL using the random index
const response = await fetch("https://ip-api.com/", {
agent: proxyAgent,
});
const data = await response.text();
console.log(data);
}
} catch (e) {
console.error(e.message);
}
})();
To iterate over the proxy list in a sequence, you can modify the code as follows:
(async () => {
try {
for (let i = 0; i < proxyUrls.length; i++) {
const proxyAgent = new HttpsProxyAgent(proxyUrls[i]);
const response = await fetch("https://ip-api.com/", { agent:proxyAgent });
const data = await response.text();
console.log(data);
}
} catch (e) {
console.error(e.message);
}
})();
Integrating proxies into Node Fetch is straightforward with the help of the https-proxy-agent package. This integration allows you to leverage Node-Fetch for web scraping projects while mitigating issues like IP blocks and geo-restrictions imposed by proxies.