我在使用HtmlUnit
解析html页面时遇到两个问题(实际上是问题)。我尝试了他们的``入门指南''以及搜索google,但没有帮助。这是我的第一个问题。
1)我想从页面中提取以下bold
标记的文本
<b class="productPrice">Five Dollars</b>
2)我想在以下结构的最后一段中提取整个文本(包括更多的跨度或链接文本,如果存在的话)
<div class="alertContainer">
<p>Hello</p>
<p>Haven't you registeret yet?</p>
<p>Registrations will close on 3 July 2012.<span>So don't wait</span></p>
</div>
您能不能请一行代码片段执行该操作?我是HtmlUnit的新手。
编辑:
HtmlUnit具有getElementByName()
和getElementById()
,因此,如果要使用类进行选择,应该使用什么?
这将是我第一个问题的答案。
参考方案
实际上,我建议您改用xpath和jtidy,像这样
import java.io.IOException;
import java.net.MalformedURLException;
import java.util.List;
import com.gargoylesoftware.htmlunit.FailingHttpStatusCodeException;
import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlItalic;
import com.gargoylesoftware.htmlunit.html.HtmlOption;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlRadioButtonInput;
import com.gargoylesoftware.htmlunit.html.HtmlSelect;
import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;
import com.gargoylesoftware.htmlunit.html.HtmlTextArea;
import com.gargoylesoftware.htmlunit.html.HtmlTextInput;
public class WebScrapper {
private static final String TEXT = "some random text here";
private static final String SWALLOW = "continental";
private static final String COLOR = "indigo2";
private static final String QUESTION = "why?";
private static final String NAME = "Leo";
/**
* @param args
* @throws IOException
* @throws MalformedURLException
* @throws FailingHttpStatusCodeException
*/
public static void main(String[] args) throws FailingHttpStatusCodeException, MalformedURLException, IOException {
//to get the HTML Xpath, download and install firefox plugin Xpather from
//http://jassage.com/xpather-1.4.5b.xpi
//
//then right-click on any part of the html and choose "show in xpather"
//
//HtmlUnit is a suite for functional web app tests (headless) with a
//built-in "browser". Very useful for screen scraping.
//
//for HtmlUnit examples and usage, try
//http://htmlunit.sourceforge.net/gettingStarted.html
//
//sometimes, the HTML is malformed, so you'll need to "clean it"
//that's why I've also added JTidy to this project
WebClient webClient = new WebClient();
HtmlPage page = webClient.getPage("http://cgi-lib.berkeley.edu/ex/simple-form.html");
// System.out.println(page.asXml());
HtmlForm form = (HtmlForm) page.getByXPath("/html/body/form").get(0);
HtmlTextInput name = form.getInputByName("name");
name.setValueAttribute(NAME);
HtmlTextInput quest = form.getInputByName("quest");
quest.setValueAttribute(QUESTION);
HtmlSelect color = form.getOneHtmlElementByAttribute("select", "name", "color");
List<HtmlOption> options = color.getOptions();
for(HtmlOption op:options){
if (op.getValueAttribute().equals(COLOR)){
op.setSelected(true);
}
}
HtmlTextArea text = form.getOneHtmlElementByAttribute("textarea", "name", "text");
text.setText(TEXT);
//swallow
HtmlRadioButtonInput swallow = form.getInputByValue(SWALLOW);
swallow.click();
HtmlSubmitInput submit = form.getInputByValue("here");
//submit
HtmlPage page2 = submit.click();
// System.out.println(page2.asXml());
String color2 = ((HtmlItalic)page2.getByXPath("//dd[1]/i").get(0)).getTextContent();
String name2 = ((HtmlItalic)page2.getByXPath("//dd[2]/i").get(0)).getTextContent();
String quest2 = ((HtmlItalic)page2.getByXPath("//dd[3]/i").get(0)).getTextContent();
String swallow2 = ((HtmlItalic)page2.getByXPath("//dd[4]/i").get(0)).getTextContent();
String text2 = ((HtmlItalic)page2.getByXPath("//dd[5]/i").get(0)).getTextContent();
System.out.println(COLOR.equals(color2)
&& NAME.equals(name2)
&& QUESTION.equals(quest2)
&& SWALLOW.equals(swallow2)
&& TEXT.equals(text2));
webClient.closeAllWindows();
}
}
Java-搜索字符串数组中的字符串 - java在Java中,我们是否有任何方法可以发现特定字符串是字符串数组的一部分。我可以避免出现一个循环。例如String [] array = {"AA","BB","CC" }; string x = "BB" 我想要一个if (some condition to tell wheth…
Java RegEx中的单词边界\ b - java我在使用\b作为Java Regex中的单词定界符时遇到困难。对于text = "/* sql statement */ INSERT INTO someTable"; Pattern.compile("(?i)\binsert\b");找不到匹配项Pattern insPtrn = Pattern.compile(&…
Java Double与BigDecimal - java我正在查看一些使用双精度变量来存储(360-359.9998779296875)结果为0.0001220703125的代码。 double变量将其存储为-1.220703125E-4。当我使用BigDecimal时,其存储为0.0001220703125。为什么将它双重存储为-1.220703125E-4? 参考方案 我不会在这里提及精度问题,而只会提及数字…
当回复有时是一个对象有时是一个数组时,如何在使用改造时解析JSON回复? - java我正在使用Retrofit来获取JSON答复。这是我实施的一部分-@GET("/api/report/list") Observable<Bills> listBill(@Query("employee_id") String employeeID); 而条例草案类是-public static class…
字符串内存分配 - java哪一个更好System.out.println("hello world"); 要么String s="hello world"; System.out.println(s); 参考方案 对于此简单示例,在内存分配方面没有区别。