Web scraping proxy setup by language - Go, Kotlin, Scala, Perl and Elixir - DataImpulse

This is a reference page rather than a tutorial: the working proxy configuration for scraping in five languages where the audience is small enough that a separate guide each would say the same thing five times. Go, Kotlin, Scala, Perl and Elixir, with the code, the library choice and the one trap that costs the most time in each.

The rule that applies to all of them is the same. Web scraping proxy setup belongs on the client object, configured once and reused, because that client is what carries the connection pool, the cookie jar and, with a sticky session, the exit address. Anything that creates a fresh client per request throws all three away.


Key Facts

  • One rule covers every language here: configure the proxy on the client object once and reuse that client, because the client is what carries the connection pool, the cookies and the exit identity.
  • Go: http.Transport with http.ProxyURL, colly for the crawler concerns. The trap is that http.Client has no default timeout.
  • Kotlin: OkHttp with a proxyAuthenticator. The trap is an authenticator that never returns null and loops on a bad password.
  • Scala: sttp with backend options. The trap is the JDK disabling Basic proxy auth on HTTPS tunnels.
  • Perl and Elixir: LWP honours HTTPS_PROXY from the environment and can override your code; in Elixir proxy options belong to the adapter, not to Req.

The 3-part rule that applies to every language here

Keep transport, parsing and scheduling in separate layers, and a broken scrape tells you where it broke.

Part Owns Symptom when it fails
1. Transport Connection, proxy, timeouts, retries, TLS Non-200 status, timeouts, TLS errors
2. Parsing Bytes to fields, and failing loudly when markup moves 200 with empty fields
3. Scheduling Concurrency per host, pacing, backoff Success that decays over minutes

Every language below implements the same three parts with different names. The transport section is where the proxy lives in all five.


Go: proxy setup, libraries and the trap to avoid

Stack: net/http with a custom http.Transport for fetching, goquery, or colly for a full crawler for parsing, chromedp when a page needs JavaScript.

proxyURL, _ := url.Parse("http://USER:[email protected]:823")
client := &http.Client{
    Transport: &http.Transport{Proxy: http.ProxyURL(proxyURL)},
    Timeout:   30 * time.Second,
}
resp, err := client.Get("https://example.com")

No default timeout. An http.Client with no Timeout waits forever. One hung host stalls a worker pool that looks healthy in every metric you have.

Unclosed bodies. Every response body must be closed or the connection never returns to the pool, and throughput quietly collapses after a few thousand requests.

colly handles the crawler concerns for you: it has a proxy switcher for rotation, per-domain rate limits and a built-in queue, so reach for it before writing your own scheduler.

Use Go when Go is already your stack, you need high concurrency with predictable memory, and you want a single static binary on the crawler box. Avoid it when the target needs a real browser for most pages, where the Python and Node ecosystems have more mature tooling.


Kotlin: proxy setup, libraries and the trap to avoid

Stack: OkHttp, or Ktor client if you are already on Ktor for fetching, jsoup for parsing, Playwright or Selenium on the JVM when rendering is required.

val proxy = Proxy(Proxy.Type.HTTP, InetSocketAddress("gw.dataimpulse.com", 823))
val client = OkHttpClient.Builder()
    .proxy(proxy)
    .proxyAuthenticator { _, response ->
        response.request.newBuilder()
            .header("Proxy-Authorization", Credentials.basic("USER", "PASS"))
            .build()
    }
    .callTimeout(Duration.ofSeconds(30))
    .build()

Authenticator loops forever on a bad password. OkHttp retries the authenticator until it gives up. Return null after the first failed attempt or a wrong credential turns into a request storm.

One client, many sessions. OkHttp shares its connection pool and cookie jar across calls. For per-session identity, build a client per session or install a cookie jar you control.

The JDK tunnelling rule that bites Java applies here too if you fall back to the JDK client: Basic proxy auth over HTTPS needs the disabled-schemes property cleared. OkHttp’s own authenticator avoids it.

Use Kotlin when you are on the JVM and want coroutines and a modern API without Java’s ceremony. Avoid it when the team has no JVM experience, since the deployment story dominates the code.


Scala: proxy setup, libraries and the trap to avoid

Stack: sttp with the backend of your choice for fetching, jsoup from Scala, or scala-scraper for parsing, Playwright on the JVM if pages render client-side.

import sttp.client3._

val backend = HttpClientSyncBackend(
  options = SttpBackendOptions.httpProxy(
    "gw.dataimpulse.com", 823, "USER", "PASS")
)

val response = basicRequest
  .get(uri"https://example.com")
  .readTimeout(30.seconds)
  .send(backend)

The backend decides everything. sttp is an interface; proxy support, timeouts and streaming come from the backend you pick. Read that backend’s docs rather than sttp’s when something behaves oddly.

JDK tunnelling again. If the backend wraps the JDK client, Basic proxy auth over HTTPS is disabled by default and needs the jdk.http.auth.tunneling.disabledSchemes property cleared.

scala-scraper wraps jsoup with a Scala API and is worth it if your parsing is more than a handful of selectors.

Use Scala when the crawler feeds a Spark or streaming pipeline that is already Scala, and backpressure matters. Avoid it when the task is a simple fetch-and-parse job with no downstream pipeline.


Perl: proxy setup, libraries and the trap to avoid

Stack: LWP::UserAgent, or Mojo::UserAgent for a modern API for fetching, HTML::TreeBuilder or Mojo::DOM for parsing, none in-process; call an external tool when rendering is needed.

use LWP::UserAgent;
my $ua = LWP::UserAgent->new(timeout => 30);
$ua->proxy(['http','https'] => 'http://USER:[email protected]:823');
my $res = $ua->get('https://example.com');
die $res->status_line unless $res->is_success;

# Mojo equivalent
my $mojo = Mojo::UserAgent->new;
$mojo->proxy->http('http://USER:[email protected]:823')
           ->https('http://USER:[email protected]:823');

HTTPS through a proxy needs the right modules. LWP needs LWP::Protocol::https and a working IO::Socket::SSL, and it also honours the HTTPS_PROXY environment variable, which silently overrides what you set in code.

Environment variables win quietly. env_proxy picks up system settings you forgot about. Set proxies explicitly and, in scheduled jobs, clear the environment so a host-level setting cannot redirect your traffic.

Mojo::UserAgent gives you non-blocking requests and a jQuery-like DOM API in one dependency, which makes it the better starting point for new Perl scrapers.

Use Perl when Perl is the language of the surrounding system, typically in long-lived ops and data tooling. Avoid it when you are starting fresh with no Perl in the stack.


Elixir: proxy setup, libraries and the trap to avoid

Stack: Req on top of Finch for fetching, Floki for parsing, Wallaby or an external browser service when rendering is required.

Req.get!("https://example.com",
  connect_options: [
    proxy: {:http, "gw.dataimpulse.com", 823, []},
    proxy_headers: [
      {"proxy-authorization",
       "Basic " <> Base.encode64("USER:PASS")}
    ]
  ],
  receive_timeout: 30_000
)
|> Map.get(:body)
|> Floki.parse_document!()

Proxy options belong to the adapter. Req passes connect_options down to Finch and Mint. If you use an older HTTPoison or hackney stack, the option shape is different, which is why copied snippets often fail silently.

A process per URL is not a plan. Spawning a Task per link starts every request at once. Task.async_stream with max_concurrency, sized per host, is the idiomatic bound.

Floki with CSS selectors keeps parsing declarative, and Req’s built-in retry and redirect handling removes most of the boilerplate people write by hand.

Use Elixir when the crawler runs inside an existing Elixir or Phoenix system and you want supervision and concurrency for free. Avoid it when you need a browser-heavy pipeline, where the ecosystem is thinner than on Node or Python.


Which language should you actually use?

The deciding factor is maintenance rather than capability. A crawler in the language your team writes daily survives the next markup change; a clever one in an unfamiliar language is abandoned at the first breakage.

Language Best fit Weakest point
Go High concurrency with predictable memory, single static binary Browser rendering tooling is thinner than Python or Node
Kotlin JVM shops that want coroutines and a modern API Deployment weight for a small job
Scala Crawlers feeding Spark or streaming pipelines Backend-dependent behaviour and a steep setup
Perl Existing ops and data tooling already in Perl No practical in-process browser
Elixir Phoenix systems wanting supervision and concurrency for free Thin browser-automation ecosystem

If your language is not here, the same three-part rule applies: our separate guides cover Java, C#, PHP, R, C++, VBA and Excel and JavaScript.


What is the same everywhere, and where are the limits?

Three things do not change with the language. Sessions own exits, so a journey across pages holds one sticky address for its whole life: the rules are in proxy rotation best practices. A refusal is a decision to diagnose rather than retry, which our guide to 403 Forbidden when scraping walks through in order. And a proxy changes only the exit address: it does not execute JavaScript, fix an inconsistent client, or make an aggressive rate acceptable.

On the rules: honour robots.txt and site terms, prefer official APIs, stay out of anything behind a login, and treat personal data as a lawful-basis question rather than a parsing one. We do not cover CAPTCHA solving or defeating access controls. General information, not legal advice: see is web scraping legal.


Frequently Asked Questions

How do I set a proxy in Go?

Parse the proxy URL with credentials, put http.ProxyURL on an http.Transport, and give the http.Client an explicit timeout, because it has none by default. Reuse that client so the connection pool and any sticky session stay intact.

Why does my Kotlin OkHttp proxy authenticator loop?

Because OkHttp keeps calling the authenticator until it succeeds or gives up. Return null after the first failed attempt so a wrong credential fails fast instead of turning into a request storm against the gateway.

Why does Basic proxy auth fail over HTTPS on the JVM?

The JDK disables Basic authentication on CONNECT tunnels by default, so an authenticated proxy works for HTTP and fails for HTTPS. Clear the jdk.http.auth.tunneling.disabledSchemes property, or use a client such as OkHttp that sends the Proxy-Authorization header itself.

Why is my Perl script ignoring the proxy I set?

LWP honours the HTTPS_PROXY and HTTP_PROXY environment variables, and env_proxy picks up host-level settings you may not know about. Set proxies explicitly in code and clear the environment in scheduled jobs so a machine setting cannot redirect your traffic.

Where do proxy options go in Elixir?

On the adapter rather than on Req itself: connect_options are passed down to Finch and Mint, which is why snippets written for older hackney-based stacks silently do nothing. Match the option shape to the adapter you actually use.


One credential, every language above

Each snippet on this page works with a DataImpulse gateway credential: residential exits at $1 per GB across 195 countries, rotating or sticky sessions, HTTP, HTTPS and SOCKS5, country targeting included. Create an account and test one target from your own client before scaling.

Related: proxies for web scraping · Crawl4AI proxy setup · web scraping use case.

Last updated: September 17, 2026.


Share article: