Java用HttpClient爬大学英语四六级考试成绩查询接口

英语四六级成绩查询地址，也是本次咱们要爬取的地址

一、Maven导包

<dependency><groupId>org.apache.httpcomponents</groupId><artifactId>httpclient</artifactId><version>4.5.12</version></dependency><dependency><groupId>mons</groupId><artifactId>commons-lang3</artifactId><version>3.9</version></dependency><dependency><groupId>com.alibaba</groupId><artifactId>fastjson</artifactId><version>1.2.68</version></dependency>

二、分析英语四六级考试成绩查询需要发送那些请求

1.第一个请求，获取验证码

①分析请求传参

http://cache./Imgs.do

这个请求一共带了三个参数

c、ik、t

后面通过分析发现参数c是写死了的，值就为CET；ik就是准考号；t是一个小于1的一个随机数，非必带的一个参数

②分析返回值

请求成功后返回的是如下这样的一个字符串，而不是直接返回的验证码图片

result.imgs(“a37b67cc9ebb4405916cb9490ddba623”);

查看上图的请求发现，他是把上个请求返回的字符通过截取拼接成一个新请求，从而获取到验证码图片并显示在页面的

分析完成后先用Postman接口测试下

测试的时候发现参数都带了，但是还是报403，经过一番排查后发现，请求头中要带Referer=http://cet./cet才行

③.代码实现

public static void main(String[] args) throws Exception {String c = "CET";String ik = "4000101";// 获取验证码地址的请求，后面写成动态的String url = "http://cache./Imgs.do?c=" + c + "&ik=" + ik;// 获得一个httpclient对象HttpGet httpget = new HttpGet(url);CloseableHttpClient httpclient = HttpClientBuilder.create().build();// 设置请求头httpget.addHeader("Referer", "http://cet./cet");CloseableHttpResponse response = null;String resultString = "";String img_url = "http://cet./imgs/";String cookie = "";try {response = httpclient.execute(httpget);if (response.getStatusLine().getStatusCode() == 200) {resultString = EntityUtils.toString(response.getEntity(), "UTF-8");// 取到括号里面的图片名称String substr = resultString.substring(13, resultString.length() - 3);System.out.println(substr);img_url += substr + ".png";// 获取图片的Base64码String imageToBase64 = NetImageToBase64(img_url);System.out.println(imageToBase64);} catch (Exception e) {e.printStackTrace();throw new InternalError(e.getMessage());}}/*** 网络图片转换Base64的方法* @param netImagePath */private static String NetImageToBase64(String netImagePath) {ByteArrayOutputStream data = new ByteArrayOutputStream();try {// 创建URLURL url = new URL(netImagePath);byte[] by = new byte[1024];// 创建链接HttpURLConnection conn = (HttpURLConnection) url.openConnection();conn.setRequestMethod("GET");conn.setConnectTimeout(5000);InputStream is = conn.getInputStream();// 将内容读取内存中int len = -1;while ((len = is.read(by)) != -1) {data.write(by, 0, len);}// 关闭流is.close();} catch (IOException e) {e.printStackTrace();}// 对字节数组Base64编码BASE64Encoder encoder = new BASE64Encoder();return encoder.encode(data.toByteArray());}

就这样我们得到验证码的Base64编码，把控制台打印出来的编码前面加上(data:image/png;base64,)复制到浏览器打开，这样就获得验证码信息了

第二个请求，查询考试成绩

查询考试成绩的接口如下

http://cache./cet/query

①分析请求传参

需要传的参数就两个

data

v

data是CET6_192_DANGCI,准考证号,姓名 拼接而成的

v就是验证码

CET6_192_DANGCI的含义：

CET4表示查询英语四级，CET6表示查询英语六级，是通过准考证的第10位来判断的，准考证第10位为1则为四级，为2则为六级(当初我这这里就吃了很大的亏)；其中192应该也是要变化的，但是目前没确定其规律

②分析返回值

result.callback({z:‘4000101’,n:‘周西’,x:‘四川大学’,s:358.00,t:0,id:‘192251016002908’,l:151,r:123,w:84,kyz:’–’,kys:’–’});

它返回的还是一个字符串，里面是Json格式的

分析完成后再用Postman接口测试下

由于有上次的经验了，咱们就在请求头中先加上Referer=http://cet./cet

但是你会发现，不管你怎么填写，都是验证码错误

后来经过不断的测试实验发现，是咱们没有携带Cookie里面一个名为verify的这个值，才导致校验验证码的时候出错了

verify这个值是在请求验证码的时候，通过响应头返回给我们的

拿到verify设置到请求头中就行了(这里我不知道是我Postman有问题，还是怎么的，一直说我验证码出错，后来我用浏览器发请求按照这样传参一下就成功了，后面用Java代码实现也没问题，很奇怪)

③.代码实现

....//在上面代码System.out.println(imageToBase64);的后面加上以下代码// 获取cookies信息String cookies = getCookie(response.getAllHeaders());String[] split = cookies.split(";");cookie = split[0];System.out.println(cookie);/*** 获取Cookie*/public static String getCookie(Header[] headers) {for (Header header : headers) {if ("Set-Cookie".equals(header.getName())) {return header.getValue();}}return null;}

下面的代码是测试获取考试成绩的

public static void main(String[] args) throws Exception {String ik = "4000101";String a = "verify=enc|f4c4fe9e2580fe1871ed5882a9aef232694e334c676d08a5a6e55fd0fe24015665759ce14873dbc35ae269589bb742bb";try {CloseableHttpResponse httpResponse = resultInquiry(ik, "周西", "nn5a",a);if (httpResponse.getStatusLine().getStatusCode() == 200) {resultString = EntityUtils.toString(httpResponse.getEntity(), "UTF-8");String skh ="(?<=\\()[^\\)]+";//用于匹配()里面的文字Pattern pattern = pile(skh);Matcher matcher = pattern.matcher(resultString);boolean is = matcher.find();if(is){System.out.print(matcher.group());Map<String,Object> jsonToMap = JSONObject.parseObject(matcher.group());System.out.println(jsonToMap);}} catch (Exception e) {e.printStackTrace();throw new InternalError(e.getMessage());}}/*** 发送第二次请求，获取考试成绩** @param ticketNumber准考证号* @param name 姓名* @param verificationCode 验证码* @param cookie* @return* @throws UnsupportedEncodingException*/public static CloseableHttpResponse resultInquiry(String ticketNumber, String name, String verificationCode, String cookie) throws UnsupportedEncodingException {String url = "http://cache./cet/query";// 获得一个httpclient对象CloseableHttpClient httpclient = HttpClients.createDefault();String idCode = idCode(ticketNumber);String data = idCode + "," + ticketNumber + "," + name;// 拼接urlurl += "?data=" + data + "&v=" + verificationCode;HttpGet httpGet = new HttpGet(url);CloseableHttpResponse response = null;// 设置请求头httpGet.addHeader("Referer", "http://cet./cet");httpGet.addHeader("Cookie", cookie);try {response = httpclient.execute(httpGet);if (response.getStatusLine().getStatusCode() == 200) {return response;}} catch (Exception e) {e.printStackTrace();throw new InternalError(e.getMessage());}return null;}/*** 通过准考证号获取data值的前缀** @param ticketNumber 准考证号* @return*/public static String idCode(String ticketNumber) {String substring = ticketNumber.substring(9, 10);//String number = ticketNumber.substring(6, 9);if ("1".equals(substring)) {return "CET4_" + 192 + "_DANGCI";} else {return "CET6_"+ 192 +"_DANGCI";}}

三、封装成工具类

由于是自己做测试的，所以上面的代码写的有些乱了，自己就封装成了一个工具类，代码如下

package top.xpcweb.demotest.utils;import com.alibaba.fastjson.JSONObject;import mons.lang3.StringUtils;import org.apache.http.Header;import org.apache.http.client.methods.CloseableHttpResponse;import org.apache.http.client.methods.HttpGet;import org.apache.http.impl.client.CloseableHttpClient;import org.apache.http.impl.client.HttpClientBuilder;import org.apache.http.impl.client.HttpClients;import org.apache.http.util.EntityUtils;import sun.misc.BASE64Encoder;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;import java.io.ByteArrayOutputStream;import java.io.IOException;import java.io.InputStream;import java.io.UnsupportedEncodingException;import .HttpURLConnection;import .URL;import java.util.HashMap;import java.util.Map;import java.util.regex.Matcher;import java.util.regex.Pattern;/*** 绩查询工具类** @author xpc* @version -06-17*/public class ResultInquiryUtils {public static final String C = "CET";public static final String IMG_URL = "http://cet./imgs/";public static final String QUERY_URL = "http://cache./cet/query";public static final String REFERER = "http://cet./cet";/*** 获取验证码** @param ik 准考证号* @param response 响应头* return 图片的Base64编码*/public static String getVerificationCode(String ik, HttpServletResponse response) {if (StringUtils.isBlank(ik)) {throw new NullPointerException("准考证号不能为null或为空。");}if (null == response) {throw new NullPointerException("响应头不能为null。");}String imageToBase64 = "";String url = "http://cache./Imgs.do?c=" + C + "&ik=" + ik;// 获得一个httpclient对象HttpGet httpget = new HttpGet(url);CloseableHttpClient httpclient = HttpClientBuilder.create().build();// 设置请求头httpget.addHeader("Referer", REFERER);CloseableHttpResponse getResponse = null;String cookie = "";try {getResponse = httpclient.execute(httpget);if (getResponse.getStatusLine().getStatusCode() == 200) {String resultString = EntityUtils.toString(getResponse.getEntity(), "UTF-8");// 取到括号里面的图片名称String substr = resultString.substring(13, resultString.length() - 3);String img_url = IMG_URL + substr + ".png";imageToBase64 = NetImageToBase64(img_url);// 获取cookies信息Header[] headers = getResponse.getHeaders("Set-Cookie");String[] split = headers[0].getValue().split(";");cookie = split[0];}response.setHeader("Eng-Cookie", cookie);} catch (Exception e) {throw new RuntimeException(e.getMessage());} finally {if (getResponse != null) {try {getResponse.close();} catch (Exception e) {throw new RuntimeException(e.getMessage());}}}return imageToBase64;}/*** 网络图片转换Base64的方法** @param netImagePath */private static String NetImageToBase64(String netImagePath) {ByteArrayOutputStream data = new ByteArrayOutputStream();try {// 创建URLURL url = new URL(netImagePath);byte[] by = new byte[1024];// 创建链接HttpURLConnection conn = (HttpURLConnection) url.openConnection();conn.setRequestMethod("GET");conn.setConnectTimeout(5000);InputStream is = conn.getInputStream();// 将内容读取内存中int len = -1;while ((len = is.read(by)) != -1) {data.write(by, 0, len);}// 关闭流is.close();} catch (IOException e) {e.printStackTrace();}// 对字节数组Base64编码BASE64Encoder encoder = new BASE64Encoder();return encoder.encode(data.toByteArray());}/*** 获取考试成绩** @param ticketNumber准考证号* @param name 姓名* @param verificationCode 验证码* @param request请求头* @return* @throws UnsupportedEncodingException*/public static Map<String, Object> resultInquiry(String ticketNumber, String name, String verificationCode,HttpServletRequest request) {if (StringUtils.isBlank(ticketNumber) || StringUtils.isBlank(name)|| StringUtils.isBlank(verificationCode) || null == request) {throw new NullPointerException("参数错误。");}String cookie = request.getHeader("Eng-Cookie");if (StringUtils.isBlank(cookie)) {throw new NullPointerException("Eng-Cookie未传。");}Map<String, Object> map = new HashMap<>();// 获得一个httpclient对象CloseableHttpClient httpclient = HttpClients.createDefault();String idCode = idCode(ticketNumber);String data = idCode + "," + ticketNumber + "," + name;// 拼接urlString url = QUERY_URL + "?data=" + data + "&v=" + verificationCode;HttpGet httpGet = new HttpGet(url);// 设置请求头httpGet.addHeader("Referer", REFERER);httpGet.addHeader("Cookie", cookie);CloseableHttpResponse response = null;try {response = httpclient.execute(httpGet);if (response.getStatusLine().getStatusCode() == 200) {String resultString = EntityUtils.toString(response.getEntity(), "UTF-8");// 正则表达式，用于匹配()里面的文字String skh = "(?<=\\()[^\\)]+";Pattern pattern = pile(skh);Matcher matcher = pattern.matcher(resultString);boolean is = matcher.find();if (is) {// 把Json数据转为mapmap = JSONObject.parseObject(matcher.group());}}} catch (Exception e) {e.printStackTrace();throw new InternalError(e.getMessage());} finally {if (response != null) {try {response.close();} catch (Exception e) {throw new RuntimeException(e.getMessage());}}}return map;}/*** 通过准考证号获取data值的前缀** @param ticketNumber 准考证号* @return*/private static String idCode(String ticketNumber) {String substring = ticketNumber.substring(9, 10);// todo 这个前缀可能每年会变if ("1".equals(substring)) {return "CET4_192_DANGCI";} else {return "CET6_192_DANGCI";}}}

顺便把Controller的代码也贴出来吧

package top.xpcweb.demotest.web;import org.springframework.stereotype.Controller;import org.springframework.web.bind.annotation.RequestMapping;import org.springframework.web.bind.annotation.ResponseBody;import top.xpcweb.demotest.utils.ResultInquiryUtils;import javax.servlet.http.HttpServletRequest;import javax.servlet.http.HttpServletResponse;import java.util.Map;/*** 绩查询Controller** @author xpc* @version -06-24*/@Controller@RequestMapping(value = "resultInquiry")public class ResultInquiryController {/*** 返回验证码名称** @param ik 准考证号*/@RequestMapping(value = "eng/verificationCode")@ResponseBodypublic String getVerificationCode(String ik, HttpServletRequest request,HttpServletResponse response) {String verificationCode = ResultInquiryUtils.getVerificationCode(ik, response);return verificationCode;}/*** 获取考试成绩接口** @param ticketNumber准考证号* @param name 姓名* @param verificationCode 验证码* @param request请求头* @param response* @return*/@RequestMapping(value = "eng/resultInquiry")@ResponseBodypublic Map<String, Object> resultInquiry(String ticketNumber, String name, String verificationCode,HttpServletRequest request, HttpServletResponse response) {Map<String, Object> map = ResultInquiryUtils.resultInquiry(ticketNumber, name, verificationCode, request);return map;}}

也欢迎大家提出我代码中的问题，或更加简便的方法

也欢迎来鱼樂圈，有兴趣的私聊我咱们一起开发此网站