HTML is the core of the web, all the pages you see on the internet are HTML, whether they are dynamically generated by JavaScript, JSP, PHP, ASP or any other web technology. Your browser actually parse HTML and render it for you. But what would you do, if you need to parse an HTML document and find some elements, tags, attributes or check if a particular element exists or not from Java program. If you have been in Java programming for some years, I am sure you have done some XML parsing work using parsers like DOM and SAX, but there is also good chance that you have not done any HTML parsing work. Ironically, there are few instances when you need to parse HTML documents from core Java application, which doesn't include Servlet and other Java web technologies.
To make the matter worse, there is no HTTP or HTML library in core JDK as well; or at least I am not aware of that. That's why when it comes to parsing an HTML file, many Java programmers had to look at Google to find out how to get the value of an HTML tag in Java.
When I needed that I was sure that there would be an open-source library that will do it for me, but didn't know that it was as wonderful and feature-rich as JSoup. It not only provides support to read and parse HTML documents but also allows you to extract any element form HTML file, their attribute, their CSS class in JQuery style, and also allows you to modify them.
You can probably do anything with an HTML document using Jsoup. In this article, we will parse and HTML file and find out the value of the title and heading tags. We will also see an example of downloading and parsing HTML from the file as well as any URL or internet by parsing Google's home page in Java.
In the second example, we will download our HTML document from web, and in third example, we will load our own sample HTML file login.html for parsing. This file is a sample HTML document that contains title tag and a div in the body that contains an HTML form.
It has input tags to capture username and password and submit and reset button for further action. It's proper HTML which can be validated i.e. all tags and attributes are properly closed. Here is how our sample HTML file look like :
HTML parsing is very simple with Jsoup, all you need to call is static method Jsoup.parse() and pass your HTML String to it. JSoup provides several overloaded parse() methods to read HTML file from String, a File, from a base URI, from an URL, and from an InputStream.
You can also specify character encoding to read HTML files correctly which is not in "UTF-8" format. Here is complete list of HTML parse methods from the JSoup library.
The parse(String html) method parses the input HTML into a new Document. In Jsoup, Document extends Element which extends Node. Also TextNode extends Node. As long as you pass in a non-null string, you're guaranteed to have a successful, sensible parse, with a Document containing (at least) a head and a body element.
Once you have a Document, you can get the data you want by calling appropriate methods in Document and its parent classes Element and Node.
Eclipse will take care of creating proper package and Java source file with same name, so absolutely less work. If you already have a Sample Java project, then it's just one step. Following Java program shows 3 examples of parsing and traversing HTML file.
In first example, we directly parse a String with HTML content, in second example we parse an HTML file downloaded from an URL, in the third example, we load and parse an HTML document from local file system.
In the first and third example, we use the parse method to get a Document object which can be queried to extract any tag value or attribute value. In the second example, we use Jsoup.connect() with, which takes care of making a connection to URL, downloading HTML and parsing it. This method also returns the Document object which can be used for further querying and getting the value of any tag or attribute.
Good thing about JSoup is that it is very robust. Jsoup HTML parser will make every attempt to create a clean parse from the HTML you provide, regardless of whether the HTML is well-formed or not. It can handle following mistakes :
unclosed tags (e.g. <p>Java <p>Scala to <p>Java</p> <p>Scala</p>)
implicit tags (e.g. a naked <td>Java is Great</td> is wrapped into a <table><tr><td>)
reliably creating the document structure (html containing a head and body, and only appropriate elements within the head)
That's all about how to parse an HTML document in Java. Jsoup is an excellent and robust open-source library that makes reading HTML document, body fragment, HTML string and directly parsing HTML content from the web extremely easy.
In this article, we learned hot to get the value of a particular HTML tag in Java, as in the first example we extracted title and value of H1 tag as text, and in the third example, we learned how to get the value of an attribute from HTML tag by extracting CSS class.
Apart from powerful jQuery style html.body().getElementsByTag("h1").text() method, which you can use to extract any HTML tag, it also provides convenience methods like Document.title() and Element.className() method to quickly get title and CSS class. Have fun with Jsoup and we will see a couple of more examples of this API soon.
Further Reading
Introduction to Spring MVC 4
RESTFul Services in Java using Jersey
Java Web Fundamentals
To make the matter worse, there is no HTTP or HTML library in core JDK as well; or at least I am not aware of that. That's why when it comes to parsing an HTML file, many Java programmers had to look at Google to find out how to get the value of an HTML tag in Java.
When I needed that I was sure that there would be an open-source library that will do it for me, but didn't know that it was as wonderful and feature-rich as JSoup. It not only provides support to read and parse HTML documents but also allows you to extract any element form HTML file, their attribute, their CSS class in JQuery style, and also allows you to modify them.
You can probably do anything with an HTML document using Jsoup. In this article, we will parse and HTML file and find out the value of the title and heading tags. We will also see an example of downloading and parsing HTML from the file as well as any URL or internet by parsing Google's home page in Java.
What is JSoup Library?
Jsoup is an open-source Java library for working with real-world HTML. It provides a very convenient API for extracting and manipulating data, using the best of DOM, CSS, and jquery-like methods. Jsoup implements the WHATWG HTML5 specification and parses HTML to the same DOM as modern browsers like Chrome and Firefox do. Here are some of the useful features of jsoup library :- Jsoup can scrape and parse HTML from a URL, file, or string
- Jsoup can find and extract data, using DOM traversal or CSS selectors
- Jsoup allows you to manipulate the HTML elements, attributes, and text
- Jsoup provides clean user-submitted content against a safe white-list, to prevent XSS attacks
- Jsoup also output tidy HTML
HTML Parsing in Java using JSoup
In this Java HTML parsing tutorial, we will see three different examples of parsing and traversing HTML documents in Java using jsoup. In the first example, we will parse an HTML String that contents all tags in form of String literal in Java.In the second example, we will download our HTML document from web, and in third example, we will load our own sample HTML file login.html for parsing. This file is a sample HTML document that contains title tag and a div in the body that contains an HTML form.
It has input tags to capture username and password and submit and reset button for further action. It's proper HTML which can be validated i.e. all tags and attributes are properly closed. Here is how our sample HTML file look like :
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd"> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=ISO-8859-1"> <title>Login Page</title> </head> <body> <div id="login" class="simple" > <form action="login.do"> Username : <input id="username" type="text" /><br> Password : <input id="password" type="password" /><br> <input id="submit" type="submit" /> <input id="reset" type="reset" /> </form> </div> </body> </html>
HTML parsing is very simple with Jsoup, all you need to call is static method Jsoup.parse() and pass your HTML String to it. JSoup provides several overloaded parse() methods to read HTML file from String, a File, from a base URI, from an URL, and from an InputStream.
You can also specify character encoding to read HTML files correctly which is not in "UTF-8" format. Here is complete list of HTML parse methods from the JSoup library.
The parse(String html) method parses the input HTML into a new Document. In Jsoup, Document extends Element which extends Node. Also TextNode extends Node. As long as you pass in a non-null string, you're guaranteed to have a successful, sensible parse, with a Document containing (at least) a head and a body element.
Once you have a Document, you can get the data you want by calling appropriate methods in Document and its parent classes Element and Node.
Java Program to parse HTML Document
Here is our complete Java program to parse an HTML String, an HTML file download from the internet, and an HTML file from the local file system. In order to run this program, you can either use Eclipse IDE or you can just use any IDE or command prompt. In Eclipse, it's very easy, just copy this code, create a new Java project, right-click on src package and paste it.Eclipse will take care of creating proper package and Java source file with same name, so absolutely less work. If you already have a Sample Java project, then it's just one step. Following Java program shows 3 examples of parsing and traversing HTML file.
In first example, we directly parse a String with HTML content, in second example we parse an HTML file downloaded from an URL, in the third example, we load and parse an HTML document from local file system.
In the first and third example, we use the parse method to get a Document object which can be queried to extract any tag value or attribute value. In the second example, we use Jsoup.connect() with, which takes care of making a connection to URL, downloading HTML and parsing it. This method also returns the Document object which can be used for further querying and getting the value of any tag or attribute.
import java.io.File; import java.io.IOException; import org.jsoup.Jsoup; import org.jsoup.nodes.Document; import org.jsoup.nodes.Element; /** * Java Program to parse/read HTML documents from File using Jsoup library. * Jsoup is an open source library which allows Java developer to parse HTML * files and extract elements, manipulate data, change style using DOM, CSS and * JQuery like method. * * @author Javin Paul */ public class HTMLParser{ public static void main(String args[]) { // Parse HTML String using JSoup library String HTMLSTring = "<!DOCTYPE html>" + "<html>" + "<head>" + "<title>JSoup Example</title>" + "</head>" + "<body>" + "<table><tr><td> <h1>HelloWorld</h1></tr>" + "</table>" + "</body>" + "</html>"; Document html = Jsoup.parse(HTMLSTring); String title = html.title(); String h1 = html.body().getElementsByTag("h1").text(); System.out.println("Input HTML String to JSoup :" + HTMLSTring); System.out.println("After parsing, Title : " + title); System.out.println("Afte parsing, Heading : " + h1); // JSoup Example 2 - Reading HTML page from URL Document doc; try { doc = Jsoup.connect("http://google.com/").get(); title = doc.title(); } catch (IOException e) { e.printStackTrace(); } System.out.println("Jsoup Can read HTML page from URL, title : " + title); // JSoup Example 3 - Parsing an HTML file in Java //Document htmlFile = Jsoup.parse("login.html", "ISO-8859-1"); // wrong Document htmlFile = null; try { htmlFile = Jsoup.parse(new File("login.html"), "ISO-8859-1"); } catch (IOException e) { // TODO Auto-generated catch block e.printStackTrace(); } // right title = htmlFile.title(); Element div = htmlFile.getElementById("login"); String cssClass = div.className(); // getting class form HTML element System.out.println("Jsoup can also parse HTML file directly"); System.out.println("title : " + title); System.out.println("class of div tag : " + cssClass); } }
Output: Input HTML String to JSoup :<!DOCTYPE html> <html><head><title>JSoup Example</title> </head><body><table><tr><td> <h1>HelloWorld</h1></tr></table></body> </html> After parsing, Title : JSoup Example Afte parsing, Heading : HelloWorld Jsoup Can read HTML page from URL, title : Google Jsoup can also parse HTML file directly title : Login Page class of div tag : simple
Good thing about JSoup is that it is very robust. Jsoup HTML parser will make every attempt to create a clean parse from the HTML you provide, regardless of whether the HTML is well-formed or not. It can handle following mistakes :
unclosed tags (e.g. <p>Java <p>Scala to <p>Java</p> <p>Scala</p>)
implicit tags (e.g. a naked <td>Java is Great</td> is wrapped into a <table><tr><td>)
reliably creating the document structure (html containing a head and body, and only appropriate elements within the head)
That's all about how to parse an HTML document in Java. Jsoup is an excellent and robust open-source library that makes reading HTML document, body fragment, HTML string and directly parsing HTML content from the web extremely easy.
In this article, we learned hot to get the value of a particular HTML tag in Java, as in the first example we extracted title and value of H1 tag as text, and in the third example, we learned how to get the value of an attribute from HTML tag by extracting CSS class.
Apart from powerful jQuery style html.body().getElementsByTag("h1").text() method, which you can use to extract any HTML tag, it also provides convenience methods like Document.title() and Element.className() method to quickly get title and CSS class. Have fun with Jsoup and we will see a couple of more examples of this API soon.
Further Reading
Introduction to Spring MVC 4
RESTFul Services in Java using Jersey
Java Web Fundamentals
This is really good article. From now on words I'l recommend this article for those who use regex for complex DOM manipulations.
ReplyDeleteKeep up the service Javin.
Very nice article, but I would like also to get HTML document from website aubmitting given cookies or making some logging into service and then checking the page as a logged in user. Can You provide such example?
ReplyDeleteJSoup is great, it is simply great. From long time we had good XML libraries and also good support of that in JDK, but not for HTML. Jsoup is filled that gap in open source space.
ReplyDeleteThis is awesome,but i want an example to find a string which contains unclosed html tags.i searched in google but i couldn't.thanks in advance..
ReplyDeletegreat examples, thanks for this.
ReplyDeletecan you please tell meusing example how to extract text from website.
ReplyDeleteCan you plzz guide me how to include web browsers in jsoup programming
ReplyDeleteis there any way to fetch html element usinf line no and its tag name ...
ReplyDelete