登录注册

当前位置：首页 > 文章列表 > Golang > Go问答 > Curl 和 Golang 中的不同请求状态（200 和 403）

Curl 和 Golang 中的不同请求状态（200 和 403）

来源：stackoverflow 2024-05-01 12:24:38 0浏览 收藏

怎么入门Golang编程？需要学习哪些知识点？这是新手们刚接触编程时常见的问题；下面golang学习网就来给大家整理分享一些知识点，希望能够给初学者一些帮助。本篇文章就来介绍《Curl 和 Golang 中的不同请求状态（200 和 403）》，涉及到，有需要的可以收藏一下

问题内容

我在本地 macbook air 上使用标准 golang http 库从网站 (https://avito.ru) 获取数据时遇到问题 - 我收到状态 403 forbidden。从浏览器打开该网站（禁用 javascript）没有问题，并且使用标准 curl 函数（curl -v https://www.avito.ru/moscow）获取数据也没有问题 - 获得状态 200 所有时间。

有一个golang代码：

package main

import (
    "crypto/tls"
    "crypto/x509"
    "flag"
    "fmt"
    "io/ioutil"
    "net/http/httputil"
    "time”
    "database/sql"
    "log"
    "net/http"

    "golang.org/x/net/http2”
    "github.com/puerkitobio/goquery"
)


func main() {

    request, err := http.newrequest("get", "https://www.avito.ru/moscow", nil)
    if err != nil {
        log.fatal(err)
    }
    request.header.set("user-agent", "mozilla/5.0 (macintosh; intel mac os x 10_15_6) applewebkit/537.36 (khtml, like gecko) chrome/86.0.4240.198 safari/537.36")
    request.header.add("accept", "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8")
    request.header.add("accept-language", "en-us")
    request.header.add("connection", "keep-alive")
    request.header.add("host", "www.avito.ru")
    fmt.println(request.header)

    // do something with the request
    client := &http.client{}
    cacert, err := ioutil.readfile("/etc/ssl/cert.pem")
    if err != nil {
        log.fatalf("reading server certificate: %s", err)
    }
    cacertpool := x509.newcertpool()
    cacertpool.appendcertsfrompem(cacert)

    var httpversion = flag.int("version", 2, "http version")

    fmt.println(string(*httpversion))
    client.transport = &http2.transport{
        tlsclientconfig: &tls.config{
            rootcas:            cacertpool,
            // insecureskipverify: true,
        },
    }

    requestdump, err := httputil.dumprequest(request, true)
    if err != nil {
        fmt.println(err)
    }
    fmt.println(string(requestdump))

    response, err := client.do(request)
    fmt.println(string(response.proto))

    if err != nil {
        log.fatal(err)
    }

    defer response.body.close()

    fmt.println(response.status)

    doc, err := goquery.newdocumentfromreader(response.body)
    if err != nil {
        fmt.println("here2")
        log.fatal(err)
    }
    pagetitle := doc.find("title").contents().text()

    fmt.println(pagetitle)
}

有印刷品：

get / http/1.1
host: www.avito.ru
user-agent: mozilla/5.0 (macintosh; intel mac os x 10_15_6) applewebkit/537.36 (khtml, like gecko) chrome/86.0.4240.198 safari/537.36
accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8
accept-language: en-us
connection: keep-alive
accept-encoding: gzip

http/2.0
map[content-type:[text/html; charset=utf-8] date:[sat, 14 nov 2020 13:14:10 gmt] server:[nginx] x-content-type-options:[nosniff] x-envoy-upstream-service-time:[14] x-xss-protection:[1; mode=block]]
403 forbidden
here3
Доступ с вашего ip-адреса временно ограничен — Авито
here4
Доступ с вашего ip-адреса временно ограничен — Авито

有 curl -v 的打印：

Alexanders-MacBook-Air:~ alexchistyakov$ curl -v https://www.avito.ru/moscow
*   Trying 185.89.12.132...
* TCP_NODELAY set
* Connected to www.avito.ru (185.89.12.132) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/cert.pem
  CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: C=RU; ST=Moscow; L=Moscow; O=Limited Liability Company KEH eCommerce; CN=*.avito.ru
*  start date: Feb  4 09:48:54 2019 GMT
*  expire date: Feb  4 09:48:54 2021 GMT
*  subjectAltName: host "www.avito.ru" matched cert's "*.avito.ru"
*  issuer: C=BE; O=GlobalSign nv-sa; CN=GlobalSign Organization Validation CA - SHA256 - G2
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7f7feb00ba00)
> GET /moscow HTTP/2
> Host: www.avito.ru
> User-Agent: curl/7.64.1
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200 
< server: nginx
< date: Sat, 14 Nov 2020 13:28:28 GMT
< content-type: text/html; charset=UTF-8
< set-cookie: u=2ke25u1o.n9oms3.9cgwbvpc4sw0; path=/; expires=Tue, 10-Nov-37 13:28:28 GMT; HttpOnly; Max-Age=536112000; secure; domain=.avito.ru
< set-cookie: v=1605360508; path=/; expires=Sat, 14-Nov-20 13:58:28 GMT; HttpOnly; Max-Age=1800; secure; domain=.avito.ru; SameSite=Lax
< cache-control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
< set-cookie: buyer_location_id=621540; expires=Sun, 14-Nov-2021 13:28:28 GMT; Max-Age=31536000; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: luri=rossiya; expires=Sun, 15-Nov-2020 13:28:28 GMT; Max-Age=86400; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: buyer_selected_search_radius4=0_general; expires=Thu, 18-Mar-3019 13:28:28 GMT; Max-Age=31504464000; path=/; domain=.avito.ru; secure; SameSite=Lax
< set-cookie: tmpBannerHash=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: tmpSideBlockHash=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: tmpSearchBottomHash=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< set-cookie: sx=H4sIAAAAAAACAw3EwQqAIAwA0H%2FZucOiJcu%2FyYiRA4UWDhL%2Fvd7hdVjSu0tpT0lVSQ5CY%2FJ%2Fh9ihQYQr31osaMskfpCIV0UUFkOuajDBCXEOuBKFDXmMD9eve2RUAAAA; expires=Sat, 21-Nov-2020 13:28:28 GMT; Max-Age=604800; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: so=1605360508; expires=Sat, 14-Nov-2020 13:43:28 GMT; Max-Age=900; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: dfp_group=52; expires=Sat, 14-Nov-2020 13:38:28 GMT; Max-Age=600; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: buyer_laas_tooltip=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: tmpSrcFromBp=ijrkns6kvj4gwc4ggwok00g8gs08oks; expires=Sat, 14-Nov-2020 14:28:28 GMT; Max-Age=3600; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: sessid=e33049edd828b91d73300175d802c306.1605360508; expires=Sun, 15-Nov-2020 13:28:28 GMT; Max-Age=86400; path=/; domain=.avito.ru; secure; HttpOnly; SameSite=Lax
< set-cookie: buyer_from_page=; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0; path=/; secure; HttpOnly; SameSite=Lax
< x-request-url: /moscow
< x-envoy-upstream-service-time: 532
< x-xss-protection: 1; mode=block
< x-content-type-options: nosniff
<