How to write a simple internet/web robot in Java?

When we say robot, yes, it is some kind of a difficult thing to do. But before we say it, let’s try writing them first.

I started writing simple internet/web robot application in Java. A robot that will scrape data from the web. A robot that will add entry to your web. And of course, you can write any kind of robots you like.

There are lot of web tools you can freely use and download from the internet. Since I use Java for my robot, I use HtmlUnit, and Selenium RC (Remote Control). These are only web tools I tested and tried so far. They have their own specific usage and purpose.

In my own experience, I use HtmlUnit for gathering and processing data from the website; and I use Selenium Remote Control for automating my own website to login and process data. These are very interesting thing I’ve learn so far.

Here is a Selenium Remote Control Java source code to open a Google site, type “Ziplok Java” in the search box and click search button:

import com.thoughtworks.selenium.*;
import junit.framework.*;

public class GoogleRobotSearch {
 private Selenium sel;

 public GoogleRobotSearch () {
  sel = new DefaultSelenium("localhost", 4444, "*firefox", "http://www.google.com");
  sel.start();
 }

 public void search() {
  sel.open("http://www.google.com/webhp?hl=en");
  sel.type("q", "Ziplok Java");
  sel.click("btnG");
  sel.waitForPageToLoad("5000");
  sel.stop();
 }

 public static void main (String args[]) {
  GoogleRobotSearch xybot = new GoogleRobotSearch ();
  xybot.search();
 }
}

And here is an equivalent Java source code written in Java with HmlUnit:

import com.gargoylesoftware.htmlunit.WebClient;
import com.gargoylesoftware.htmlunit.html.HtmlForm;
import com.gargoylesoftware.htmlunit.html.HtmlPage;
import com.gargoylesoftware.htmlunit.html.HtmlSubmitInput;
import com.gargoylesoftware.htmlunit.html.HtmlTextInput;

public class GoogleRobotSearch {
 private String bUrl;

 public GoogleRobotSearch (String url) throws Exception {
  bUrl = url;
 }

 public void search () throws Exception {
  WebClient wb = new WebClient ();
  HtmlPage p = (HtmlPage) wb.getPage(bUrl);

  HtmlForm f = p.getFormByName("f");
  HtmlTextInput text = (HtmlTextInput) f.getInputByName("q");
  HtmlSubmitInput submit = (HtmlSubmitInput) f.getInputByName("btnG");
  text.setValueAttribute("Ziplok Java");

  HtmlPage resultPage = (HtmlPage) submit.click();
  System.out.println(resultPage.asText());
 }

 public static void main (String args[]) throws Exception {
  GoogleRobotSearch xyro = new GoogleRobotSearch ("http://www.google.com/");
  xyro.search ();
 }
}

That’s it! It is just easy to write an internet/web robot.

How will you type a text into the input file element in Selenium?

The common problem in Selenium is typing a text into the input element (<input type=”file” />). And this is the common problem I read from most of the forums and I also find it difficult to solve.

I am just a newbie in Selenium and I do not know how to handle problems using this web testing tool. I’ve been Googling in a week just to find an answer. I was lucky because I found one and it really works.

Here are the possible solutions to your problem:

• Use *chrome environment type rather than *firefox or *iexplore.
• Set your singed.applets.codebase_principal_support to true in your Mozilla configuration.

I’m crossing my finger on that. Try it and tell me if that works in your machine.

How to update a file in Java?

I’ve been reading books in Java just to find a code on how to update a file just to append a text in it. A lot of code fragments on Java File available but it just give you how to write a file alone, and or how to read a file alone. This article will help you to create at the same time append a text to an existing file.

Here’s a sample of simple and short Java code to create (if not created) and append a file if existing.

public void createOrAppendFile (File f, String text) throws IOException {
 BufferedWriter bw = new BufferedWriter (new FileWriter (File, true));
 bw.write (text);
 bw.newLine();
 bw.flush();
 bw.close();
}

How to setup a headless application in Linux?

I am a newbie to Linux environment and do not know every commands available inside the shell.

This entry will explain and demonstrate to all newbies out there and to those who have not know to setup a headless application in Linux and for those who want to setup headless application in Linux.

Install the following in your Linux distro if you have not install them yet.
1. Xvfb
2. startx
3. import from ImageMagick Tools

Let’s try to run Firefox without a visible display.
1. Start Xvfb.
startx -- `which Xvfb` :1 -screen 0 1024x768x24
2. start Xvfb running on :1 with a screen size of 1024×768 and 24bits/pixel color depth.
DISPLAY=:1 firefox
3. Take a screenshot.
DISPLAY=:1 import -window root firefox.png
Thank you to Jordan Sissel for his blog regarding this problem.

How will I post a Java source code in WordPress?

I should say to myself, “Welcome to the world of blogging!”.

Yes, this is my first time to post a source code in my blog. And I believe that most of the blog sites today can support different programming languages. And I’ll show you one on how to display a Java source code in your blog.

Take note: This sample is effective only in WordPress.

Here is the view of my sample Java source code:

// HelloWorld.java
public class HelloWorld {
   public static void main (String args[]) {
      System.out.println ("Hello, World!");
   }
}

Here is how will you do it:

//  HelloWorld.java
public class HelloWorld {
   public static void main (String args[]) {
      System.out.println ("Hello, World!");
   }
}

I think I am ready now to write more blog entries on Java and other programming problems. Check this out always.

Click here for more information about posting source code.

Hello, world!

Hello, world!