Connecting Tech Pros Worldwide Help | Site Map

mySQL to XML - very slow - need some criticism

bergy's Avatar
Member
 
Join Date: Mar 2007
Posts: 90
#1: Sep 21 '07
Preface:
I have some data being inserting into a mySQL table every 5 minutes. Currently, there are about 275 rows. Each row has 26 (TEXT) columns, the data in those columns is no longer than 10-15 characters.

I am using the following code to turn this data into XML which is then, through a servlet, output on a webserver. With 275 rows, this is taking around 3-4 minutes to generate the XML file. I am worried that in a week from now when the data is over 2,000 rows it will exponentially decrease in speed. Can someone give me a better route so this will run faster?

Expand|Select|Wrap|Line Numbers
  1.         String xml = "<?xml version=\"1.0\" encoding=\"utf-8\"?>\n\n";
  2.       String SQLCommand = "select * from "+getDataId()+";";
  3.       try{
  4.           Statement stmt;     
  5.           ResultSet rs;
  6.           String url = "jdbc:mysql://"+getIBoxIp()+":"+getDbPort()+"/"+getDbName();          
  7.           Connection con = DriverManager.getConnection(url,getDbUsername(), getDbPassword());                    
  8.               stmt = con.createStatement(ResultSet.TYPE_SCROLL_INSENSITIVE, ResultSet.CONCUR_READ_ONLY);
  9.               rs = stmt.executeQuery(SQLCommand);
  10.               while(rs.next()){
  11.                 xml += "\n<row>";
  12.                 ResultSetMetaData md = rs.getMetaData();
  13.                 for(int i = 1; i < md.getColumnCount() + 1; i++){
  14.                     xml += "\n\t<"+md.getColumnName(i)+">"+rs.getString(i)+"</"+md.getColumnName(i)+">";
  15.                 }
  16.                 xml += "\n</row>\n";
  17.               }
  18.               System.out.println("@@mysql> XMLSQLCommand > " + SQLCommand);          
  19.           con.close();      
  20.           return xml;
  21.       }catch(Exception ex)
  22.       {
  23.           System.out.println("@@mysql> SQLCommand Failed > " + SQLCommand + "\n@@mysql> Exception > " + ex.getMessage() + "");
  24.           return xml;
  25.       }
Again, with 275 rows, I'm getting about 250 KB of data in the resulting XML file with a 3-5 minute parsing/generation time.

Thanks in advance!!!
JosAH's Avatar
Expert
 
Join Date: Mar 2007
Posts: 10,611
#2: Sep 21 '07

re: mySQL to XML - very slow - need some criticism


Don't simply add strings together as in 'xml+= somethingElse', most certainly
not in loops. Use StringBuffers or even better StringBuilders for that operation.

kind regards,

Jos
bergy's Avatar
Member
 
Join Date: Mar 2007
Posts: 90
#3: Sep 21 '07

re: mySQL to XML - very slow - need some criticism


Thanks Jos, I'm a java newbie - being forced into it because of the platform I'm writing on - much appreciated!!
JosAH's Avatar
Expert
 
Join Date: Mar 2007
Posts: 10,611
#4: Sep 21 '07

re: mySQL to XML - very slow - need some criticism


Quote:

Originally Posted by bergy

Thanks Jos, I'm a java newbie - being forced into it because of the platform I'm writing on - much appreciated!!

You're welcome of course; you're doing fine except for all those String concatenations;
simply add all those String parts to a builder or buffer and extract the String when
you're done looping over all those rows and columns.

kind regards,

Jos
bergy's Avatar
Member
 
Join Date: Mar 2007
Posts: 90
#5: Sep 21 '07

re: mySQL to XML - very slow - need some criticism


Yep - this worked of course - turning 4 minutes in to 4 milliseconds :) Thanks again!
JosAH's Avatar
Expert
 
Join Date: Mar 2007
Posts: 10,611
#6: Sep 21 '07

re: mySQL to XML - very slow - need some criticism


Quote:

Originally Posted by bergy

Yep - this worked of course - turning 4 minutes in to 4 milliseconds :) Thanks again!

Don't trust that 4 milliseconds figure; when the hotspot/jit compilers wakes up
it could be even better (or not, depending on your database query). But you
certainly did notice the speed up because of those builders ;-) Try it with a
couple of ~ 10,000 records and see what happens. I'm sure we can say "problem solved".

kind regards,

Jos
Reply