JAVA: 抓取需要登录的页面, curl page with auth, How do I connect to a URL using Basic authentication?

ConnectToUrlUsingBasicAuthentication.java

package test;

import java.io.IOException;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.net.MalformedURLException;
import java.net.URL;
import java.net.URLConnection;

import org.apache.commons.codec.binary.Base64;

public class ConnectToUrlUsingBasicAuthentication {

	public static void main(String[] args) {

		try {
			String webPage = "http://192.168.1.1";
			String name = "admin";
			String password = "admin";

			String authString = name + ":" + password;
			System.out.println("auth string: " + authString);
			byte[] authEncBytes = Base64.encodeBase64(authString.getBytes());
			String authStringEnc = new String(authEncBytes);
			System.out.println("Base64 encoded auth string: " + authStringEnc);

			URL url = new URL(webPage);
			URLConnection urlConnection = url.openConnection();
			urlConnection.setRequestProperty("Authorization", "Basic " + authStringEnc);
			InputStream is = urlConnection.getInputStream();
			InputStreamReader isr = new InputStreamReader(is);

			int numCharsRead;
			char[] charArray = new char[1024];
			StringBuffer sb = new StringBuffer();
			while ((numCharsRead = isr.read(charArray)) > 0) {
				sb.append(charArray, 0, numCharsRead);
			}
			String result = sb.toString();

			System.out.println("*** BEGIN ***");
			System.out.println(result);
			System.out.println("*** END ***");
		} catch (MalformedURLException e) {
			e.printStackTrace();
		} catch (IOException e) {
			e.printStackTrace();
		}
	}

}

The execution of the ConnectToUrlUsingBasicAuthentication class is shown below.

JAVA: 抓取需要登录的页面, curl page with auth, How do I connect to a URL using Basic authentication?
JAVA: 抓取需要登录的页面, curl page with auth, How do I connect to a URL using Basic authentication?

结合jackson抓取json页面:

    public static JsonNode curl(String url, String username, String password) {
        JsonNode node = null;

        try {
            String authString = username + ":" + password;
            byte[] authEncBytes = Base64.encodeBase64(authString.getBytes());
            String authStringEnc = new String(authEncBytes);

            URLConnection urlConnection = new URL(url).openConnection();
            urlConnection.setRequestProperty("Authorization", "Basic " + authStringEnc);

            ObjectMapper mapper = new ObjectMapper();
            node = mapper.readTree(urlConnection.getInputStream());
        } catch (IOException e) {
            e.printStackTrace();
        }

        return node;
    }

参看:JAVA: 使用Jackson解析JSON, 生成JSON, 反序列化和序列化, Jackson 实现JSON数据与Java对象相互转换, 详解入门(附项目源码)

核心部分就是:

ObjectMapper mapper = new ObjectMapper();

JsonNode node = mapper.readTree(connection.getInputStream());

// Grab statusCode with node.get("StatusCode").intValue()
// Grab CustomerName with node.get("CustomerName").textValue()

Jackson中readTree用法:读取json并解析成JsonNode树

项目中需要解析一个200M的json目录文件,然后根据需要读取field

从路径为jsonPath的文件中读取了json字符串,然后转换成JsonNode

import java.io.IOException;  
import java.nio.file.Files;  
import java.nio.file.Path;  
  
  
import com.fasterxml.jackson.core.JsonProcessingException;  
import com.fasterxml.jackson.databind.JsonNode;  
import com.fasterxml.jackson.databind.ObjectMapper;  
  
public class ReadJsonFile {  
      
    private ObjectMapper mapper = new ObjectMapper();  
      
    public JsonNode readAsTree(Path jsonPath) throws Exception, IOException {  
        JsonNode jsonNode =  mapper.readTree(Files.newInputStream(jsonPath));  
        return jsonNode;  
    }  
  
}

把JsonNode对象转成json字符串写入到某个文件中,也可以是其他类型的对象

import java.io.IOException;  
import java.nio.file.Files;  
import java.nio.file.Path;  
  
import com.fasterxml.jackson.databind.JsonNode;  
import com.fasterxml.jackson.databind.ObjectMapper;  
import com.fasterxml.jackson.databind.SequenceWriter;  
  
public class WriteJsonFile {  
  
    private ObjectMapper mapper = new ObjectMapper();  
  
    public void writeJsonFile(JsonNode rootNode, Path outputPath) throws IOException {  
        SequenceWriter writer = mapper.writerWithDefaultPrettyPrinter()  
                .writeValues(Files.newOutputStream(outputPath));  
        writer.write(rootNode);  
        writer.close();  
    }  
      
}

如果是一个JsonNode数组,比如这里的arrayNode,使用arrayNode.elements();读取数组中每个node

如果不是JsonNode数组arrayNode.elements();返回arrayNode的values

public void handleElements(JsonNode arrayNode) {  
    Iterator<JsonNode> realElements = arrayNode.elements();  
    while (realElements.hasNext()) {  
        JsonNode realElement = realElements.next();  
        System.out.println(realElement.toString());  
    }  
}

jsonNode的fieldNames就是该jsonNode的所有的key值

public void traverseJson(JsonNode jsonNode) throws IOException {  
    Iterator<String> fieldNames = jsonNode.fieldNames();  
    while (fieldNames.hasNext()) {  
        String fieldName = fieldNames.next();  
        System.out.println(fieldName);  
    }  
}

类似Map的Entry方式遍历某个JsonNode的key和value, value可能是字符串也可能是子jsonNode,但如果value是jsonNode数组的话,是无法读取的

public void traverseJson(JsonNode jsonNode) {  
    Iterator<Entry<String, JsonNode>> jsonNodes = jsonNode.fields();  
    while (jsonNodes.hasNext()) {  
        Entry<String, JsonNode> node = jsonNodes.next();  
        System.out.println(node.getKey());  
        System.out.println(node.getValue().toString());  
    }  
}

取出所有key值为fieldName的JsonNode的List

public void traverseJson(JsonNode jsonNode) {  
    List<JsonNode> jsonNodes = jsonNode.findParents("fieldName");  
    System.out.println(jsonNodes.size());  
}

取出所有key值为fieldName对应的value,这些values就是JsonNode的List,但是如果value中包含子jsonNode并且子jsonNode的key值也为fieldName,是无法捕获到并加入list的

public void traverseJson(JsonNode jsonNode) {  
    List<JsonNode> jsonNodes = jsonNode.findValues("fieldName");  
    System.out.println(jsonNodes.size());  
}

取出key值为fieldName的一个value,匹配到一个就停止搜索,并且返回value

public void traverseJson(JsonNode jsonNode) {  
    JsonNode node1 = jsonNode.findValue("fieldName");  
    JsonNode node2 = jsonNode.findPath("fieldName");  
}

node1和node2的区别在于,如果没有匹配到任何key值为fieldName,node1为null,node2为空JsonNode

类似于

List<String> node1 = null;  
List<String> node2 = new ArrayList<>();

如果value为String,可以这样读取,其他类型可以jsonNode.findValue(“name”).asInt();

public void traverseJson(JsonNode jsonNode) {  
    //{"name" : "xiaoLi"}  
    String name = jsonNode.findValue("name").asText();  
    System.out.println(name);  
}

如果是JDK1.8的话,可以这样遍历JsonNode的子节点

ublic void traverseJson(JsonNode jsonNode) {  
    jsonNode.forEach((JsonNode node)->{  
        System.out.println(node.toString());  
    });  
}

 

本文:JAVA: 抓取需要登录的页面, curl page with auth, How do I connect to a URL using Basic authentication?