RE: Query regarding execution of multiple xqueries with s axon API

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Dasgupta, Subhojit (Subhojit)** CTR **
Hi Michael

I have developed a Java API that internally uses saxon 8.5.1 methods to
execute xqueries from an external file.As you had suggested earlier , i am
now loading the big XML file (~90MB) using buildDocument() only once.Also
all my xqueries are defined as functions in the query.xq file.Which query to
execute is being controlled from the Java API. While executing the xqueries
i found that the execution time is much more than what i had expected for
every xquery in my file.Is it possible that for each of these queries the
xml document is getting loaded every time inadvertently?

My pseudo code is as follows:

        final Configuration config = new Configuration();
        final StaticQueryContext sqc = new StaticQueryContext(config);

        final XQueryExpression exp = sqc.compileQuery(new
FileReader("./query.xq"));
      DynamicQueryContext dynamicContext = new DynamicQueryContext(config);
       
        dynamicContext.setContextNode(sqc.buildDocument(new
StreamSource("./Mydata.xml")));

         String s;
        BufferedReader nr = new BufferedReader(new
FileReader("querynames.xml"));
           try {
                        while ( (s = nr.readLine()) != null ) {
                        String sp = "<qnames>";
                        String ep = "</qnames>";
                        if (s.startsWith(sp)) {
                             String str = s.substring(6);
                             String qname = str.replaceAll(ep, "");
                        final UserFunction fn =
exp.getStaticContext().getUserDefinedFunction("http://www.myfunc.com/query",
"QueryMain", 1);
                        final Value[] arglist = new Value[1];
                      arglist[0] = new StringValue(qname);
        final Controller controller = exp.getController();
                        startTime = System.currentTimeMillis();
            final ValueRepresentation result = fn.call(arglist,
controller);
                                endTime = System.currentTimeMillis();
       
                                time = (endTime - startTime)/1000.0;
                                TimeInmins = time/60.0;
       
                                System.out.println("Queryme :" + qname + ","
+ " Result : " + result + " Execution Time in minutes : " + TimeInmins);
                        }
                   }
                } catch (Exception e) {
                        System.out.println("Error reading line: "
+e.getMessage());
                        e.printStackTrace();
                  }
                }

I would like to know if it is ok to use the method controller which seems to
be deprecated now.I must tell you that a xqyery which is taking more than 1
min to execute should have taken less than a minute ideally.Thus my total
execution time has gone up drastically.

I would appreciate any help in this regard.

Thanks
Subhojit



-----Original Message-----
From: [hidden email]
[mailto:[hidden email]]On Behalf Of Michael Kay
Sent: Thursday, November 17, 2005 8:49 PM
To: 'Dasgupta, Subhojit (Subhojit)** CTR **'
Cc: [hidden email]
Subject: RE: [saxon] Query regarding execution of multiple xqueries with
saxon API


Could I ask you again - please copy your messages to the list so there is an
archive.

I would suggest you put each of your "queries" into a separate function, and
compile them all as part of a single module. Then have a query parameter
(external global variable) that defines which query you want to execute, and
the main body of your query should test this variable and call one of the
functions as appropriate.


declare variable $q external;
declare function my:q1() {
...
}
declare function my:q2() {
...
}
if ($q="q1") then my:q1()
else if ($q="q2") then my:q2()
...
else error()



Separate compilation of library modules is something I would like to
implement at some stage in the future.

Michael Kay
http://www.saxonica.com/ 

> -----Original Message-----
> From: Dasgupta, Subhojit (Subhojit)** CTR **
> [mailto:[hidden email]]
> Sent: 17 November 2005 14:26
> To: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with saxon API
>
> Hi Michael
>
> Thanks a lot  for the valuable inputs.I do agree with you on
> this point that
> i have been executing multiple queries as separate
> expressions within a
> single query expressed as a list. Ideally i would like to
> execute individual
> queries one after another against the same xml data file
> which is also huge.
> The issue here is that the xqueries that i have written make
> use of a common
> library module(functions defined in the same module) which i
> imported within
> the xquery file.The reason for this is that i wanted to
> import the module
> once for all the queries in the static context and not load
> it repeatedly
> for all the queries.I apologise if i am getting unclear here.
>
> I can think of a way whereby for each of the xqueries i would
> call compile
> query and then execute them in the dynamic context.But that
> would also need
> the external library to be loaded every time. Please correct
> me if i am
> wrong.I wanted to avoid this totally. I cant think of a way
> whereby i can
> import the library module completely independent of the xquery file!.
>
> So it would be great if you can suggest me some way of importing this
> library module only once for all the queries.Also i would
> like to mention
> that the xml document is being loaded only once using buildDocument().
>
> I appreciate your effort and valuable time spent.
>
> Best Regards
> Subhojit
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of
> Michael Kay
> Sent: Wednesday, November 16, 2005 9:37 PM
> To: [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> saxon API
>
>
> I'm not sure where your difficulty lies. Perhaps it would be
> easier if you
> showed your code.
>
> Firstly, I don't understand why you want to create multiple
> queries from the
> same .xq file. Why not create one query and execute it multiple times?
>
> Secondly, if you do need to create multiple queries, there is
> no difficulty
> doing so. You just call compileQuery() more than once. You
> can either use
> the same StaticContext or a different StaticContext for each query.
>
> There's also no problem in parsing the source document once
> and then using
> it repeatedly. Call the buildDocument() method in
> StaticQueryContext once to
> build the DocumentInfo, then supply this DocumentInfo as the
> context item in
> the dynamic context for each query evaluation.
>
> I hope this helps.
>
> Michael Kay
> http://www.saxonica.com/ 
>
> > -----Original Message-----
> > From: [hidden email]
> > [mailto:[hidden email]] On Behalf Of
> > Dasgupta, Subhojit (Subhojit)** CTR **
> > Sent: 16 November 2005 14:23
> > To: '[hidden email]'
> > Subject: [saxon] Query regarding execution of multiple
> > xqueries with saxon API
> >
> > Hi
> >
> > I am using Saxon B 8.5.1 version and i need to execute
> > multiple xqueries in
> > my application. I have written a java API that compiles the
> > xqueries that i
> > have and the library module  in a static context and then
> > execute those
> > xqueries in a dynamic context.Also it loads the xml file in
> > the dynamic
> > context. For this i used the standard  functions provided by
> > saxon (like
> > compileQuery(), evaluateSingle() etc). The issue that i am
> > facing is that
> > the above works fine when i am using a single xquery.But in
> > my application
> > i need to execute multiple xqueries from the same  .xq file
> > in the same
> > context and collect some performance statistics for all those
> > xqueries (viz
> > execution time for each xquery etc). Also i cannot afford to
> > load the XML
> > file more than once as it as huge!!.
> >
> > With the setup i have as mentioned here, i am unable to execute
> > multiple xqueries. Does Saxon provide an alternate way of
> > achieving the same
> > and collect performance data for each and every xquery that
> i execute?
> >
> > I am looking forward to some help in this regard as to how
> > multiple xqueries
> > can be created in saxon without creating  a context for every
> > new query from
> > the xquery file.
> >
> > Best Regards
> > Subhojit
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc.  Get
> Certified Today
> > Register for a JBoss Training Course.  Free Certification Exam
> > for All Training Attendees Through End of 2005. For more info visit:
> > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > _______________________________________________
> > saxon-help mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/saxon-help
> >
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
> Register for a JBoss Training Course.  Free Certification Exam
> for All Training Attendees Through End of 2005. For more info visit:
> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>




-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
Register for a JBoss Training Course.  Free Certification Exam
for All Training Attendees Through End of 2005. For more info visit:
http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Dasgupta, Subhojit (Subhojit)** CTR **
Hi Michael

Just to throw some more light into my problem , below is one of the sample
queries which i am trying to execute:

declare namespace ns = "http://www.xyz.com";

every $x in ($ns:<xpath>) satisfies (every $y in ($x[not((ns:f1() >
ns:f2()))]) satisfies (false()));

When i executed the above query against a 90MB XML file, it took 1.44 mins
for execution.The loading time for me has been 1 min 7 secs consistently for
the setup i have.

Would it be possible to bring this time to less than a minute or few
seconds?Will the query plan anlysis do any good here?

Kindly help.

Thanks
Subhojit



-----Original Message-----
From: [hidden email]
[mailto:[hidden email]]On Behalf Of Dasgupta,
Subhojit (Subhojit)** CTR **
Sent: Thursday, December 01, 2005 6:37 PM
To: '[hidden email]'
Cc: '[hidden email]'
Subject: RE: [saxon] Query regarding execution of multiple xqueries with
s axon API


Hi Michael

I have developed a Java API that internally uses saxon 8.5.1 methods to
execute xqueries from an external file.As you had suggested earlier , i am
now loading the big XML file (~90MB) using buildDocument() only once.Also
all my xqueries are defined as functions in the query.xq file.Which query to
execute is being controlled from the Java API. While executing the xqueries
i found that the execution time is much more than what i had expected for
every xquery in my file.Is it possible that for each of these queries the
xml document is getting loaded every time inadvertently?

My pseudo code is as follows:

        final Configuration config = new Configuration();
        final StaticQueryContext sqc = new StaticQueryContext(config);

        final XQueryExpression exp = sqc.compileQuery(new
FileReader("./query.xq"));
      DynamicQueryContext dynamicContext = new DynamicQueryContext(config);
       
        dynamicContext.setContextNode(sqc.buildDocument(new
StreamSource("./Mydata.xml")));

         String s;
        BufferedReader nr = new BufferedReader(new
FileReader("querynames.xml"));
           try {
                        while ( (s = nr.readLine()) != null ) {
                        String sp = "<qnames>";
                        String ep = "</qnames>";
                        if (s.startsWith(sp)) {
                             String str = s.substring(6);
                             String qname = str.replaceAll(ep, "");
                        final UserFunction fn =
exp.getStaticContext().getUserDefinedFunction("http://www.myfunc.com/query",
"QueryMain", 1);
                        final Value[] arglist = new Value[1];
                      arglist[0] = new StringValue(qname);
        final Controller controller = exp.getController();
                        startTime = System.currentTimeMillis();
            final ValueRepresentation result = fn.call(arglist,
controller);
                                endTime = System.currentTimeMillis();
       
                                time = (endTime - startTime)/1000.0;
                                TimeInmins = time/60.0;
       
                                System.out.println("Queryme :" + qname + ","
+ " Result : " + result + " Execution Time in minutes : " + TimeInmins);
                        }
                   }
                } catch (Exception e) {
                        System.out.println("Error reading line: "
+e.getMessage());
                        e.printStackTrace();
                  }
                }

I would like to know if it is ok to use the method controller which seems to
be deprecated now.I must tell you that a xqyery which is taking more than 1
min to execute should have taken less than a minute ideally.Thus my total
execution time has gone up drastically.

I would appreciate any help in this regard.

Thanks
Subhojit



-----Original Message-----
From: [hidden email]
[mailto:[hidden email]]On Behalf Of Michael Kay
Sent: Thursday, November 17, 2005 8:49 PM
To: 'Dasgupta, Subhojit (Subhojit)** CTR **'
Cc: [hidden email]
Subject: RE: [saxon] Query regarding execution of multiple xqueries with
saxon API


Could I ask you again - please copy your messages to the list so there is an
archive.

I would suggest you put each of your "queries" into a separate function, and
compile them all as part of a single module. Then have a query parameter
(external global variable) that defines which query you want to execute, and
the main body of your query should test this variable and call one of the
functions as appropriate.


declare variable $q external;
declare function my:q1() {
...
}
declare function my:q2() {
...
}
if ($q="q1") then my:q1()
else if ($q="q2") then my:q2()
...
else error()



Separate compilation of library modules is something I would like to
implement at some stage in the future.

Michael Kay
http://www.saxonica.com/ 

> -----Original Message-----
> From: Dasgupta, Subhojit (Subhojit)** CTR **
> [mailto:[hidden email]]
> Sent: 17 November 2005 14:26
> To: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with saxon API
>
> Hi Michael
>
> Thanks a lot  for the valuable inputs.I do agree with you on
> this point that
> i have been executing multiple queries as separate
> expressions within a
> single query expressed as a list. Ideally i would like to
> execute individual
> queries one after another against the same xml data file
> which is also huge.
> The issue here is that the xqueries that i have written make
> use of a common
> library module(functions defined in the same module) which i
> imported within
> the xquery file.The reason for this is that i wanted to
> import the module
> once for all the queries in the static context and not load
> it repeatedly
> for all the queries.I apologise if i am getting unclear here.
>
> I can think of a way whereby for each of the xqueries i would
> call compile
> query and then execute them in the dynamic context.But that
> would also need
> the external library to be loaded every time. Please correct
> me if i am
> wrong.I wanted to avoid this totally. I cant think of a way
> whereby i can
> import the library module completely independent of the xquery file!.
>
> So it would be great if you can suggest me some way of importing this
> library module only once for all the queries.Also i would
> like to mention
> that the xml document is being loaded only once using buildDocument().
>
> I appreciate your effort and valuable time spent.
>
> Best Regards
> Subhojit
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of
> Michael Kay
> Sent: Wednesday, November 16, 2005 9:37 PM
> To: [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> saxon API
>
>
> I'm not sure where your difficulty lies. Perhaps it would be
> easier if you
> showed your code.
>
> Firstly, I don't understand why you want to create multiple
> queries from the
> same .xq file. Why not create one query and execute it multiple times?
>
> Secondly, if you do need to create multiple queries, there is
> no difficulty
> doing so. You just call compileQuery() more than once. You
> can either use
> the same StaticContext or a different StaticContext for each query.
>
> There's also no problem in parsing the source document once
> and then using
> it repeatedly. Call the buildDocument() method in
> StaticQueryContext once to
> build the DocumentInfo, then supply this DocumentInfo as the
> context item in
> the dynamic context for each query evaluation.
>
> I hope this helps.
>
> Michael Kay
> http://www.saxonica.com/ 
>
> > -----Original Message-----
> > From: [hidden email]
> > [mailto:[hidden email]] On Behalf Of
> > Dasgupta, Subhojit (Subhojit)** CTR **
> > Sent: 16 November 2005 14:23
> > To: '[hidden email]'
> > Subject: [saxon] Query regarding execution of multiple
> > xqueries with saxon API
> >
> > Hi
> >
> > I am using Saxon B 8.5.1 version and i need to execute
> > multiple xqueries in
> > my application. I have written a java API that compiles the
> > xqueries that i
> > have and the library module  in a static context and then
> > execute those
> > xqueries in a dynamic context.Also it loads the xml file in
> > the dynamic
> > context. For this i used the standard  functions provided by
> > saxon (like
> > compileQuery(), evaluateSingle() etc). The issue that i am
> > facing is that
> > the above works fine when i am using a single xquery.But in
> > my application
> > i need to execute multiple xqueries from the same  .xq file
> > in the same
> > context and collect some performance statistics for all those
> > xqueries (viz
> > execution time for each xquery etc). Also i cannot afford to
> > load the XML
> > file more than once as it as huge!!.
> >
> > With the setup i have as mentioned here, i am unable to execute
> > multiple xqueries. Does Saxon provide an alternate way of
> > achieving the same
> > and collect performance data for each and every xquery that
> i execute?
> >
> > I am looking forward to some help in this regard as to how
> > multiple xqueries
> > can be created in saxon without creating  a context for every
> > new query from
> > the xquery file.
> >
> > Best Regards
> > Subhojit
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc.  Get
> Certified Today
> > Register for a JBoss Training Course.  Free Certification Exam
> > for All Training Attendees Through End of 2005. For more info visit:
> > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > _______________________________________________
> > saxon-help mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/saxon-help
> >
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
> Register for a JBoss Training Course.  Free Certification Exam
> for All Training Attendees Through End of 2005. For more info visit:
> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>




-------------------------------------------------------
This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
Register for a JBoss Training Course.  Free Certification Exam
for All Training Attendees Through End of 2005. For more info visit:
http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help


-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

Re: Query regarding execution of multiple xqueries with s axon API

Wolfgang Hoschek-2
Most of this is probably related to the underlying XML parser as well  
as (fruitless) garbage collection attempts by the VM during building  
the tree. Make sure your heap is large enough (e.g. java -Xms400m -
Xmx1000m). If you are on JDK <= 1.4, try to run with a recent XML  
parser such as xerces-2.7.1. Or simple use 1.5. Make sure to use the  
java -server option. Finally, consider evaluating a binary  
alternative such as PTree or bnux.

Wolfgang.

On Dec 1, 2005, at 6:27 AM, Dasgupta, Subhojit (Subhojit)** CTR **  
wrote:

> Hi Michael
>
> Just to throw some more light into my problem , below is one of the  
> sample
> queries which i am trying to execute:
>
> declare namespace ns = "http://www.xyz.com";
>
> every $x in ($ns:<xpath>) satisfies (every $y in ($x[not((ns:f1() >
> ns:f2()))]) satisfies (false()));
>
> When i executed the above query against a 90MB XML file, it took  
> 1.44 mins
> for execution.The loading time for me has been 1 min 7 secs  
> consistently for
> the setup i have.
>
> Would it be possible to bring this time to less than a minute or few
> seconds?Will the query plan anlysis do any good here?
>
> Kindly help.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of Dasgupta,
> Subhojit (Subhojit)** CTR **
> Sent: Thursday, December 01, 2005 6:37 PM
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple xqueries  
> with
> s axon API
>
>
> Hi Michael
>
> I have developed a Java API that internally uses saxon 8.5.1  
> methods to
> execute xqueries from an external file.As you had suggested  
> earlier , i am
> now loading the big XML file (~90MB) using buildDocument() only  
> once.Also
> all my xqueries are defined as functions in the query.xq file.Which  
> query to
> execute is being controlled from the Java API. While executing the  
> xqueries
> i found that the execution time is much more than what i had  
> expected for
> every xquery in my file.Is it possible that for each of these  
> queries the
> xml document is getting loaded every time inadvertently?
>
> My pseudo code is as follows:
>
>         final Configuration config = new Configuration();
>         final StaticQueryContext sqc = new StaticQueryContext(config);
>
>     final XQueryExpression exp = sqc.compileQuery(new
> FileReader("./query.xq"));
>       DynamicQueryContext dynamicContext = new DynamicQueryContext
> (config);
>
>     dynamicContext.setContextNode(sqc.buildDocument(new
> StreamSource("./Mydata.xml")));
>
>      String s;
>         BufferedReader nr = new BufferedReader(new
> FileReader("querynames.xml"));
>        try {
>             while ( (s = nr.readLine()) != null ) {
>             String sp = "<qnames>";
>             String ep = "</qnames>";
>             if (s.startsWith(sp)) {
>                  String str = s.substring(6);
>                  String qname = str.replaceAll(ep, "");
>             final UserFunction fn =
> exp.getStaticContext().getUserDefinedFunction("http://
> www.myfunc.com/query",
> "QueryMain", 1);
>             final Value[] arglist = new Value[1];
>               arglist[0] = new StringValue(qname);
>                 final Controller controller = exp.getController();
>             startTime = System.currentTimeMillis();
>                     final ValueRepresentation result = fn.call
> (arglist,
> controller);
>                 endTime = System.currentTimeMillis();
>
>                 time = (endTime - startTime)/1000.0;
>                 TimeInmins = time/60.0;
>
>                 System.out.println("Queryme :" + qname + ","
> + " Result : " + result + " Execution Time in minutes : " +  
> TimeInmins);
>             }
>            }
>         } catch (Exception e) {
>             System.out.println("Error reading line: "
> +e.getMessage());
>             e.printStackTrace();
>           }
>         }
>
> I would like to know if it is ok to use the method controller which  
> seems to
> be deprecated now.I must tell you that a xqyery which is taking  
> more than 1
> min to execute should have taken less than a minute ideally.Thus my  
> total
> execution time has gone up drastically.
>
> I would appreciate any help in this regard.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of Michael  
> Kay
> Sent: Thursday, November 17, 2005 8:49 PM
> To: 'Dasgupta, Subhojit (Subhojit)** CTR **'
> Cc: [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple xqueries  
> with
> saxon API
>
>
> Could I ask you again - please copy your messages to the list so  
> there is an
> archive.
>
> I would suggest you put each of your "queries" into a separate  
> function, and
> compile them all as part of a single module. Then have a query  
> parameter
> (external global variable) that defines which query you want to  
> execute, and
> the main body of your query should test this variable and call one  
> of the
> functions as appropriate.
>
>
> declare variable $q external;
> declare function my:q1() {
> ...
> }
> declare function my:q2() {
> ...
> }
> if ($q="q1") then my:q1()
> else if ($q="q2") then my:q2()
> ...
> else error()
>
>
>
> Separate compilation of library modules is something I would like to
> implement at some stage in the future.
>
> Michael Kay
> http://www.saxonica.com/
>
>
>> -----Original Message-----
>> From: Dasgupta, Subhojit (Subhojit)** CTR **
>> [mailto:[hidden email]]
>> Sent: 17 November 2005 14:26
>> To: '[hidden email]'
>> Subject: RE: [saxon] Query regarding execution of multiple
>> xqueries with saxon API
>>
>> Hi Michael
>>
>> Thanks a lot  for the valuable inputs.I do agree with you on
>> this point that
>> i have been executing multiple queries as separate
>> expressions within a
>> single query expressed as a list. Ideally i would like to
>> execute individual
>> queries one after another against the same xml data file
>> which is also huge.
>> The issue here is that the xqueries that i have written make
>> use of a common
>> library module(functions defined in the same module) which i
>> imported within
>> the xquery file.The reason for this is that i wanted to
>> import the module
>> once for all the queries in the static context and not load
>> it repeatedly
>> for all the queries.I apologise if i am getting unclear here.
>>
>> I can think of a way whereby for each of the xqueries i would
>> call compile
>> query and then execute them in the dynamic context.But that
>> would also need
>> the external library to be loaded every time. Please correct
>> me if i am
>> wrong.I wanted to avoid this totally. I cant think of a way
>> whereby i can
>> import the library module completely independent of the xquery file!.
>>
>> So it would be great if you can suggest me some way of importing this
>> library module only once for all the queries.Also i would
>> like to mention
>> that the xml document is being loaded only once using buildDocument
>> ().
>>
>> I appreciate your effort and valuable time spent.
>>
>> Best Regards
>> Subhojit
>>
>> -----Original Message-----
>> From: [hidden email]
>> [mailto:[hidden email]]On Behalf Of
>> Michael Kay
>> Sent: Wednesday, November 16, 2005 9:37 PM
>> To: [hidden email]
>> Subject: RE: [saxon] Query regarding execution of multiple
>> xqueries with
>> saxon API
>>
>>
>> I'm not sure where your difficulty lies. Perhaps it would be
>> easier if you
>> showed your code.
>>
>> Firstly, I don't understand why you want to create multiple
>> queries from the
>> same .xq file. Why not create one query and execute it multiple  
>> times?
>>
>> Secondly, if you do need to create multiple queries, there is
>> no difficulty
>> doing so. You just call compileQuery() more than once. You
>> can either use
>> the same StaticContext or a different StaticContext for each query.
>>
>> There's also no problem in parsing the source document once
>> and then using
>> it repeatedly. Call the buildDocument() method in
>> StaticQueryContext once to
>> build the DocumentInfo, then supply this DocumentInfo as the
>> context item in
>> the dynamic context for each query evaluation.
>>
>> I hope this helps.
>>
>> Michael Kay
>> http://www.saxonica.com/
>>
>>
>>> -----Original Message-----
>>> From: [hidden email]
>>> [mailto:[hidden email]] On Behalf Of
>>> Dasgupta, Subhojit (Subhojit)** CTR **
>>> Sent: 16 November 2005 14:23
>>> To: '[hidden email]'
>>> Subject: [saxon] Query regarding execution of multiple
>>> xqueries with saxon API
>>>
>>> Hi
>>>
>>> I am using Saxon B 8.5.1 version and i need to execute
>>> multiple xqueries in
>>> my application. I have written a java API that compiles the
>>> xqueries that i
>>> have and the library module  in a static context and then
>>> execute those
>>> xqueries in a dynamic context.Also it loads the xml file in
>>> the dynamic
>>> context. For this i used the standard  functions provided by
>>> saxon (like
>>> compileQuery(), evaluateSingle() etc). The issue that i am
>>> facing is that
>>> the above works fine when i am using a single xquery.But in
>>> my application
>>> i need to execute multiple xqueries from the same  .xq file
>>> in the same
>>> context and collect some performance statistics for all those
>>> xqueries (viz
>>> execution time for each xquery etc). Also i cannot afford to
>>> load the XML
>>> file more than once as it as huge!!.
>>>
>>>     With the setup i have as mentioned here, i am unable to execute
>>> multiple xqueries. Does Saxon provide an alternate way of
>>> achieving the same
>>> and collect performance data for each and every xquery that
>>>
>> i execute?
>>
>>>
>>> I am looking forward to some help in this regard as to how
>>> multiple xqueries
>>> can be created in saxon without creating  a context for every
>>> new query from
>>> the xquery file.
>>>
>>> Best Regards
>>> Subhojit
>>>
>>>
>>>
>>> -------------------------------------------------------
>>> This SF.Net email is sponsored by the JBoss Inc.  Get
>>>
>> Certified Today
>>
>>> Register for a JBoss Training Course.  Free Certification Exam
>>> for All Training Attendees Through End of 2005. For more info visit:
>>> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
>>> _______________________________________________
>>> saxon-help mailing list
>>> [hidden email]
>>> https://lists.sourceforge.net/lists/listinfo/saxon-help
>>>
>>>
>>
>>
>>
>>
>> -------------------------------------------------------
>> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
>> Register for a JBoss Training Course.  Free Certification Exam
>> for All Training Attendees Through End of 2005. For more info visit:
>> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
>> _______________________________________________
>> saxon-help mailing list
>> [hidden email]
>> https://lists.sourceforge.net/lists/listinfo/saxon-help
>>
>>
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
> Register for a JBoss Training Course.  Free Certification Exam
> for All Training Attendees Through End of 2005. For more info visit:
> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through  
> log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD  
> SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep through  
> log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  DOWNLOAD  
> SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

Bug in processing command-line options?

Manuel Baehnisch
Hi,

I've been trying to use saxon from the command line by initiating a
certain template call via the '-it template' parameter.

When ever I tried to start the Transformation starting at the 'main'
template:
java net.sf.saxon.Transform -it main File.xml Sheet.xsl

I got the response:
Bad param=value pair on command line: Sheet.xsl

Since File.xml was just a dummy file I accidently tried it without and
suddenly it worked (java net.sf.saxon.Transform -it main Sheet.xsl).
Just out of curiousity: Is that really intended?

Thanks.
Manuel Baehnisch






-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Michael Kay
In reply to this post by Dasgupta, Subhojit (Subhojit)** CTR **
A further reply: a processing speed of about 1Mb/sec is actually not that
unreasonable. One can sometimes do better, it depends on the system
configuration. A lot depends on what's in that XPath that you haven't shown
us - in particular, does it include any predicates, and how selective are
they?

Michael Kay
http://www.saxonica.com/

> -----Original Message-----
> From: Dasgupta, Subhojit (Subhojit)** CTR **
> [mailto:[hidden email]]
> Sent: 01 December 2005 14:27
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with s axon API
>
> Hi Michael
>
> Just to throw some more light into my problem , below is one
> of the sample
> queries which i am trying to execute:
>
> declare namespace ns = "http://www.xyz.com";
>
> every $x in ($ns:<xpath>) satisfies (every $y in ($x[not((ns:f1() >
> ns:f2()))]) satisfies (false()));
>
> When i executed the above query against a 90MB XML file, it
> took 1.44 mins
> for execution.The loading time for me has been 1 min 7 secs
> consistently for
> the setup i have.
>
> Would it be possible to bring this time to less than a minute or few
> seconds?Will the query plan anlysis do any good here?
>
> Kindly help.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of Dasgupta,
> Subhojit (Subhojit)** CTR **
> Sent: Thursday, December 01, 2005 6:37 PM
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> s axon API
>
>
> Hi Michael
>
> I have developed a Java API that internally uses saxon 8.5.1
> methods to
> execute xqueries from an external file.As you had suggested
> earlier , i am
> now loading the big XML file (~90MB) using buildDocument()
> only once.Also
> all my xqueries are defined as functions in the query.xq
> file.Which query to
> execute is being controlled from the Java API. While
> executing the xqueries
> i found that the execution time is much more than what i had
> expected for
> every xquery in my file.Is it possible that for each of these
> queries the
> xml document is getting loaded every time inadvertently?
>
> My pseudo code is as follows:
>
>         final Configuration config = new Configuration();
>         final StaticQueryContext sqc = new StaticQueryContext(config);
>
> final XQueryExpression exp = sqc.compileQuery(new
> FileReader("./query.xq"));
>       DynamicQueryContext dynamicContext = new
> DynamicQueryContext(config);
>
> dynamicContext.setContextNode(sqc.buildDocument(new
> StreamSource("./Mydata.xml")));
>
> String s;
>         BufferedReader nr = new BufferedReader(new
> FileReader("querynames.xml"));
>   try {
> while ( (s = nr.readLine()) != null ) {
> String sp = "<qnames>";
> String ep = "</qnames>";
> if (s.startsWith(sp)) {
>     String str = s.substring(6);
>     String qname = str.replaceAll(ep, "");
> final UserFunction fn =
> exp.getStaticContext().getUserDefinedFunction("http://www.myfu
> nc.com/query",
> "QueryMain", 1);
> final Value[] arglist = new Value[1];
>      arglist[0] = new StringValue(qname);
>         final Controller controller =
> exp.getController();
> startTime = System.currentTimeMillis();
>             final ValueRepresentation result =
> fn.call(arglist,
> controller);
> endTime = System.currentTimeMillis();
>
> time = (endTime - startTime)/1000.0;
> TimeInmins = time/60.0;
>
> System.out.println("Queryme :"
> + qname + ","
> + " Result : " + result + " Execution Time in minutes : " +
> TimeInmins);
> }
>   }
> } catch (Exception e) {
> System.out.println("Error reading line: "
> +e.getMessage());
> e.printStackTrace();
>  }
> }
>
> I would like to know if it is ok to use the method controller
> which seems to
> be deprecated now.I must tell you that a xqyery which is
> taking more than 1
> min to execute should have taken less than a minute
> ideally.Thus my total
> execution time has gone up drastically.
>
> I would appreciate any help in this regard.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of
> Michael Kay
> Sent: Thursday, November 17, 2005 8:49 PM
> To: 'Dasgupta, Subhojit (Subhojit)** CTR **'
> Cc: [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> saxon API
>
>
> Could I ask you again - please copy your messages to the list
> so there is an
> archive.
>
> I would suggest you put each of your "queries" into a
> separate function, and
> compile them all as part of a single module. Then have a
> query parameter
> (external global variable) that defines which query you want
> to execute, and
> the main body of your query should test this variable and
> call one of the
> functions as appropriate.
>
>
> declare variable $q external;
> declare function my:q1() {
> ...
> }
> declare function my:q2() {
> ...
> }
> if ($q="q1") then my:q1()
> else if ($q="q2") then my:q2()
> ...
> else error()
>
>
>
> Separate compilation of library modules is something I would like to
> implement at some stage in the future.
>
> Michael Kay
> http://www.saxonica.com/ 
>
> > -----Original Message-----
> > From: Dasgupta, Subhojit (Subhojit)** CTR **
> > [mailto:[hidden email]]
> > Sent: 17 November 2005 14:26
> > To: '[hidden email]'
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with saxon API
> >
> > Hi Michael
> >
> > Thanks a lot  for the valuable inputs.I do agree with you on
> > this point that
> > i have been executing multiple queries as separate
> > expressions within a
> > single query expressed as a list. Ideally i would like to
> > execute individual
> > queries one after another against the same xml data file
> > which is also huge.
> > The issue here is that the xqueries that i have written make
> > use of a common
> > library module(functions defined in the same module) which i
> > imported within
> > the xquery file.The reason for this is that i wanted to
> > import the module
> > once for all the queries in the static context and not load
> > it repeatedly
> > for all the queries.I apologise if i am getting unclear here.
> >
> > I can think of a way whereby for each of the xqueries i would
> > call compile
> > query and then execute them in the dynamic context.But that
> > would also need
> > the external library to be loaded every time. Please correct
> > me if i am
> > wrong.I wanted to avoid this totally. I cant think of a way
> > whereby i can
> > import the library module completely independent of the
> xquery file!.
> >
> > So it would be great if you can suggest me some way of
> importing this
> > library module only once for all the queries.Also i would
> > like to mention
> > that the xml document is being loaded only once using
> buildDocument().
> >
> > I appreciate your effort and valuable time spent.
> >
> > Best Regards
> > Subhojit
> >
> > -----Original Message-----
> > From: [hidden email]
> > [mailto:[hidden email]]On Behalf Of
> > Michael Kay
> > Sent: Wednesday, November 16, 2005 9:37 PM
> > To: [hidden email]
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with
> > saxon API
> >
> >
> > I'm not sure where your difficulty lies. Perhaps it would be
> > easier if you
> > showed your code.
> >
> > Firstly, I don't understand why you want to create multiple
> > queries from the
> > same .xq file. Why not create one query and execute it
> multiple times?
> >
> > Secondly, if you do need to create multiple queries, there is
> > no difficulty
> > doing so. You just call compileQuery() more than once. You
> > can either use
> > the same StaticContext or a different StaticContext for each query.
> >
> > There's also no problem in parsing the source document once
> > and then using
> > it repeatedly. Call the buildDocument() method in
> > StaticQueryContext once to
> > build the DocumentInfo, then supply this DocumentInfo as the
> > context item in
> > the dynamic context for each query evaluation.
> >
> > I hope this helps.
> >
> > Michael Kay
> > http://www.saxonica.com/ 
> >
> > > -----Original Message-----
> > > From: [hidden email]
> > > [mailto:[hidden email]] On Behalf Of
> > > Dasgupta, Subhojit (Subhojit)** CTR **
> > > Sent: 16 November 2005 14:23
> > > To: '[hidden email]'
> > > Subject: [saxon] Query regarding execution of multiple
> > > xqueries with saxon API
> > >
> > > Hi
> > >
> > > I am using Saxon B 8.5.1 version and i need to execute
> > > multiple xqueries in
> > > my application. I have written a java API that compiles the
> > > xqueries that i
> > > have and the library module  in a static context and then
> > > execute those
> > > xqueries in a dynamic context.Also it loads the xml file in
> > > the dynamic
> > > context. For this i used the standard  functions provided by
> > > saxon (like
> > > compileQuery(), evaluateSingle() etc). The issue that i am
> > > facing is that
> > > the above works fine when i am using a single xquery.But in
> > > my application
> > > i need to execute multiple xqueries from the same  .xq file
> > > in the same
> > > context and collect some performance statistics for all those
> > > xqueries (viz
> > > execution time for each xquery etc). Also i cannot afford to
> > > load the XML
> > > file more than once as it as huge!!.
> > >
> > > With the setup i have as mentioned here, i am unable to execute
> > > multiple xqueries. Does Saxon provide an alternate way of
> > > achieving the same
> > > and collect performance data for each and every xquery that
> > i execute?
> > >
> > > I am looking forward to some help in this regard as to how
> > > multiple xqueries
> > > can be created in saxon without creating  a context for every
> > > new query from
> > > the xquery file.
> > >
> > > Best Regards
> > > Subhojit
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by the JBoss Inc.  Get
> > Certified Today
> > > Register for a JBoss Training Course.  Free Certification Exam
> > > for All Training Attendees Through End of 2005. For more
> info visit:
> > > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > > _______________________________________________
> > > saxon-help mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/saxon-help
> > >
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc.  Get
> Certified Today
> > Register for a JBoss Training Course.  Free Certification Exam
> > for All Training Attendees Through End of 2005. For more info visit:
> > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > _______________________________________________
> > saxon-help mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/saxon-help
> >
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
> Register for a JBoss Training Course.  Free Certification Exam
> for All Training Attendees Through End of 2005. For more info visit:
> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep
> through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  
> DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Michael Kay
In reply to this post by Dasgupta, Subhojit (Subhojit)** CTR **
I don't think there's any problem with your calling Java code, I suspect the
problem is with your query. You could confirm this by running the same query
from the command line.

When investigating this kind of performance problem, it's useful to explore
how the execution time changes as the data file increases in size. In
particular, it's useful to know whether the execution time varies linearly
or quadratically with the data size.

Your query as written looks rather odd. I would like to see the complete
query, that is, including the xpath expression and the functions f1 and f2.
Without the complete query, I can't tell what the optimizer is likely to do
with it. In fact, Saxon doesn't try very hard to optimize "some" and "every"
expressions, because they aren't used all that often; you might find it
helps to rewrite the expression by hand. You can always rewrite

every $x in X satisfies E

as

not(exists(X[not(E)]))

provided you take care to replace references to the $x with references to
the context item.

Michael Kay
http://www.saxonica.com/

> -----Original Message-----
> From: Dasgupta, Subhojit (Subhojit)** CTR **
> [mailto:[hidden email]]
> Sent: 01 December 2005 14:27
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with s axon API
>
> Hi Michael
>
> Just to throw some more light into my problem , below is one
> of the sample
> queries which i am trying to execute:
>
> declare namespace ns = "http://www.xyz.com";
>
> every $x in ($ns:<xpath>) satisfies (every $y in ($x[not((ns:f1() >
> ns:f2()))]) satisfies (false()));
>
> When i executed the above query against a 90MB XML file, it
> took 1.44 mins
> for execution.The loading time for me has been 1 min 7 secs
> consistently for
> the setup i have.
>
> Would it be possible to bring this time to less than a minute or few
> seconds?Will the query plan anlysis do any good here?
>
> Kindly help.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of Dasgupta,
> Subhojit (Subhojit)** CTR **
> Sent: Thursday, December 01, 2005 6:37 PM
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> s axon API
>
>
> Hi Michael
>
> I have developed a Java API that internally uses saxon 8.5.1
> methods to
> execute xqueries from an external file.As you had suggested
> earlier , i am
> now loading the big XML file (~90MB) using buildDocument()
> only once.Also
> all my xqueries are defined as functions in the query.xq
> file.Which query to
> execute is being controlled from the Java API. While
> executing the xqueries
> i found that the execution time is much more than what i had
> expected for
> every xquery in my file.Is it possible that for each of these
> queries the
> xml document is getting loaded every time inadvertently?
>
> My pseudo code is as follows:
>
>         final Configuration config = new Configuration();
>         final StaticQueryContext sqc = new StaticQueryContext(config);
>
> final XQueryExpression exp = sqc.compileQuery(new
> FileReader("./query.xq"));
>       DynamicQueryContext dynamicContext = new
> DynamicQueryContext(config);
>
> dynamicContext.setContextNode(sqc.buildDocument(new
> StreamSource("./Mydata.xml")));
>
> String s;
>         BufferedReader nr = new BufferedReader(new
> FileReader("querynames.xml"));
>   try {
> while ( (s = nr.readLine()) != null ) {
> String sp = "<qnames>";
> String ep = "</qnames>";
> if (s.startsWith(sp)) {
>     String str = s.substring(6);
>     String qname = str.replaceAll(ep, "");
> final UserFunction fn =
> exp.getStaticContext().getUserDefinedFunction("http://www.myfu
> nc.com/query",
> "QueryMain", 1);
> final Value[] arglist = new Value[1];
>      arglist[0] = new StringValue(qname);
>         final Controller controller =
> exp.getController();
> startTime = System.currentTimeMillis();
>             final ValueRepresentation result =
> fn.call(arglist,
> controller);
> endTime = System.currentTimeMillis();
>
> time = (endTime - startTime)/1000.0;
> TimeInmins = time/60.0;
>
> System.out.println("Queryme :"
> + qname + ","
> + " Result : " + result + " Execution Time in minutes : " +
> TimeInmins);
> }
>   }
> } catch (Exception e) {
> System.out.println("Error reading line: "
> +e.getMessage());
> e.printStackTrace();
>  }
> }
>
> I would like to know if it is ok to use the method controller
> which seems to
> be deprecated now.I must tell you that a xqyery which is
> taking more than 1
> min to execute should have taken less than a minute
> ideally.Thus my total
> execution time has gone up drastically.
>
> I would appreciate any help in this regard.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of
> Michael Kay
> Sent: Thursday, November 17, 2005 8:49 PM
> To: 'Dasgupta, Subhojit (Subhojit)** CTR **'
> Cc: [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> saxon API
>
>
> Could I ask you again - please copy your messages to the list
> so there is an
> archive.
>
> I would suggest you put each of your "queries" into a
> separate function, and
> compile them all as part of a single module. Then have a
> query parameter
> (external global variable) that defines which query you want
> to execute, and
> the main body of your query should test this variable and
> call one of the
> functions as appropriate.
>
>
> declare variable $q external;
> declare function my:q1() {
> ...
> }
> declare function my:q2() {
> ...
> }
> if ($q="q1") then my:q1()
> else if ($q="q2") then my:q2()
> ...
> else error()
>
>
>
> Separate compilation of library modules is something I would like to
> implement at some stage in the future.
>
> Michael Kay
> http://www.saxonica.com/ 
>
> > -----Original Message-----
> > From: Dasgupta, Subhojit (Subhojit)** CTR **
> > [mailto:[hidden email]]
> > Sent: 17 November 2005 14:26
> > To: '[hidden email]'
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with saxon API
> >
> > Hi Michael
> >
> > Thanks a lot  for the valuable inputs.I do agree with you on
> > this point that
> > i have been executing multiple queries as separate
> > expressions within a
> > single query expressed as a list. Ideally i would like to
> > execute individual
> > queries one after another against the same xml data file
> > which is also huge.
> > The issue here is that the xqueries that i have written make
> > use of a common
> > library module(functions defined in the same module) which i
> > imported within
> > the xquery file.The reason for this is that i wanted to
> > import the module
> > once for all the queries in the static context and not load
> > it repeatedly
> > for all the queries.I apologise if i am getting unclear here.
> >
> > I can think of a way whereby for each of the xqueries i would
> > call compile
> > query and then execute them in the dynamic context.But that
> > would also need
> > the external library to be loaded every time. Please correct
> > me if i am
> > wrong.I wanted to avoid this totally. I cant think of a way
> > whereby i can
> > import the library module completely independent of the
> xquery file!.
> >
> > So it would be great if you can suggest me some way of
> importing this
> > library module only once for all the queries.Also i would
> > like to mention
> > that the xml document is being loaded only once using
> buildDocument().
> >
> > I appreciate your effort and valuable time spent.
> >
> > Best Regards
> > Subhojit
> >
> > -----Original Message-----
> > From: [hidden email]
> > [mailto:[hidden email]]On Behalf Of
> > Michael Kay
> > Sent: Wednesday, November 16, 2005 9:37 PM
> > To: [hidden email]
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with
> > saxon API
> >
> >
> > I'm not sure where your difficulty lies. Perhaps it would be
> > easier if you
> > showed your code.
> >
> > Firstly, I don't understand why you want to create multiple
> > queries from the
> > same .xq file. Why not create one query and execute it
> multiple times?
> >
> > Secondly, if you do need to create multiple queries, there is
> > no difficulty
> > doing so. You just call compileQuery() more than once. You
> > can either use
> > the same StaticContext or a different StaticContext for each query.
> >
> > There's also no problem in parsing the source document once
> > and then using
> > it repeatedly. Call the buildDocument() method in
> > StaticQueryContext once to
> > build the DocumentInfo, then supply this DocumentInfo as the
> > context item in
> > the dynamic context for each query evaluation.
> >
> > I hope this helps.
> >
> > Michael Kay
> > http://www.saxonica.com/ 
> >
> > > -----Original Message-----
> > > From: [hidden email]
> > > [mailto:[hidden email]] On Behalf Of
> > > Dasgupta, Subhojit (Subhojit)** CTR **
> > > Sent: 16 November 2005 14:23
> > > To: '[hidden email]'
> > > Subject: [saxon] Query regarding execution of multiple
> > > xqueries with saxon API
> > >
> > > Hi
> > >
> > > I am using Saxon B 8.5.1 version and i need to execute
> > > multiple xqueries in
> > > my application. I have written a java API that compiles the
> > > xqueries that i
> > > have and the library module  in a static context and then
> > > execute those
> > > xqueries in a dynamic context.Also it loads the xml file in
> > > the dynamic
> > > context. For this i used the standard  functions provided by
> > > saxon (like
> > > compileQuery(), evaluateSingle() etc). The issue that i am
> > > facing is that
> > > the above works fine when i am using a single xquery.But in
> > > my application
> > > i need to execute multiple xqueries from the same  .xq file
> > > in the same
> > > context and collect some performance statistics for all those
> > > xqueries (viz
> > > execution time for each xquery etc). Also i cannot afford to
> > > load the XML
> > > file more than once as it as huge!!.
> > >
> > > With the setup i have as mentioned here, i am unable to execute
> > > multiple xqueries. Does Saxon provide an alternate way of
> > > achieving the same
> > > and collect performance data for each and every xquery that
> > i execute?
> > >
> > > I am looking forward to some help in this regard as to how
> > > multiple xqueries
> > > can be created in saxon without creating  a context for every
> > > new query from
> > > the xquery file.
> > >
> > > Best Regards
> > > Subhojit
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by the JBoss Inc.  Get
> > Certified Today
> > > Register for a JBoss Training Course.  Free Certification Exam
> > > for All Training Attendees Through End of 2005. For more
> info visit:
> > > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > > _______________________________________________
> > > saxon-help mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/saxon-help
> > >
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc.  Get
> Certified Today
> > Register for a JBoss Training Course.  Free Certification Exam
> > for All Training Attendees Through End of 2005. For more info visit:
> > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > _______________________________________________
> > saxon-help mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/saxon-help
> >
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
> Register for a JBoss Training Course.  Free Certification Exam
> for All Training Attendees Through End of 2005. For more info visit:
> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep
> through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  
> DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Bug in processing command-line options?

Michael Kay
In reply to this post by Manuel Baehnisch
 
> Since File.xml was just a dummy file I accidently tried it without and
> suddenly it worked (java net.sf.saxon.Transform -it main Sheet.xsl).
> Just out of curiousity: Is that really intended?

Yes (and it's documented this way). The main purpose of the -it option is to
allow you to run stylesheets that don't have a principal source document,
therefore with this option you don't specify one. (It's possible via the
Java API to have both an initial template and a principal source document,
but this combination isn't supported on the command line).

Michael Kay
http://www.saxonica.com/




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Dasgupta, Subhojit (Subhojit)** CTR **
In reply to this post by Dasgupta, Subhojit (Subhojit)** CTR **
Hi Michael

Thanks for the inputs.

Some interesting findings from my side.I executed the same query i mentioned
earlier from command line now with the java -server option and i am able to
get some optimisation.The query finished execution in less than 20
secs.However when executed using the API it is taking now 1 min 20 sec
(reduction in 20 secs .It was 1.44 before).Also i have observed that the
loading time has come down after i have used this -server option .

But still i am not able to get the optimisation in the API way as i got in
the command line.Does it mean that the
command line execution is more efficient in terms of performance
optimisation? Is it possible that the xml document is being loaded every
time per query in the API scheme of things?

Also is there something else that i can do to optimise the java code
further.

Thanks
Subhojit




-----Original Message-----
From: Michael Kay [mailto:[hidden email]]
Sent: Friday, December 02, 2005 5:52 AM
To: 'Dasgupta, Subhojit (Subhojit)** CTR **';
[hidden email]
Subject: RE: [saxon] Query regarding execution of multiple xqueries with
s axon API


I don't think there's any problem with your calling Java code, I suspect the
problem is with your query. You could confirm this by running the same query
from the command line.

When investigating this kind of performance problem, it's useful to explore
how the execution time changes as the data file increases in size. In
particular, it's useful to know whether the execution time varies linearly
or quadratically with the data size.

Your query as written looks rather odd. I would like to see the complete
query, that is, including the xpath expression and the functions f1 and f2.
Without the complete query, I can't tell what the optimizer is likely to do
with it. In fact, Saxon doesn't try very hard to optimize "some" and "every"
expressions, because they aren't used all that often; you might find it
helps to rewrite the expression by hand. You can always rewrite

every $x in X satisfies E

as

not(exists(X[not(E)]))

provided you take care to replace references to the $x with references to
the context item.

Michael Kay
http://www.saxonica.com/

> -----Original Message-----
> From: Dasgupta, Subhojit (Subhojit)** CTR **
> [mailto:[hidden email]]
> Sent: 01 December 2005 14:27
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with s axon API
>
> Hi Michael
>
> Just to throw some more light into my problem , below is one
> of the sample
> queries which i am trying to execute:
>
> declare namespace ns = "http://www.xyz.com";
>
> every $x in ($ns:<xpath>) satisfies (every $y in ($x[not((ns:f1() >
> ns:f2()))]) satisfies (false()));
>
> When i executed the above query against a 90MB XML file, it
> took 1.44 mins
> for execution.The loading time for me has been 1 min 7 secs
> consistently for
> the setup i have.
>
> Would it be possible to bring this time to less than a minute or few
> seconds?Will the query plan anlysis do any good here?
>
> Kindly help.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of Dasgupta,
> Subhojit (Subhojit)** CTR **
> Sent: Thursday, December 01, 2005 6:37 PM
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> s axon API
>
>
> Hi Michael
>
> I have developed a Java API that internally uses saxon 8.5.1
> methods to
> execute xqueries from an external file.As you had suggested
> earlier , i am
> now loading the big XML file (~90MB) using buildDocument()
> only once.Also
> all my xqueries are defined as functions in the query.xq
> file.Which query to
> execute is being controlled from the Java API. While
> executing the xqueries
> i found that the execution time is much more than what i had
> expected for
> every xquery in my file.Is it possible that for each of these
> queries the
> xml document is getting loaded every time inadvertently?
>
> My pseudo code is as follows:
>
>         final Configuration config = new Configuration();
>         final StaticQueryContext sqc = new StaticQueryContext(config);
>
> final XQueryExpression exp = sqc.compileQuery(new
> FileReader("./query.xq"));
>       DynamicQueryContext dynamicContext = new
> DynamicQueryContext(config);
>
> dynamicContext.setContextNode(sqc.buildDocument(new
> StreamSource("./Mydata.xml")));
>
> String s;
>         BufferedReader nr = new BufferedReader(new
> FileReader("querynames.xml"));
>   try {
> while ( (s = nr.readLine()) != null ) {
> String sp = "<qnames>";
> String ep = "</qnames>";
> if (s.startsWith(sp)) {
>     String str = s.substring(6);
>     String qname = str.replaceAll(ep, "");
> final UserFunction fn =
> exp.getStaticContext().getUserDefinedFunction("http://www.myfu
> nc.com/query",
> "QueryMain", 1);
> final Value[] arglist = new Value[1];
>      arglist[0] = new StringValue(qname);
>         final Controller controller =
> exp.getController();
> startTime = System.currentTimeMillis();
>             final ValueRepresentation result =
> fn.call(arglist,
> controller);
> endTime = System.currentTimeMillis();
>
> time = (endTime - startTime)/1000.0;
> TimeInmins = time/60.0;
>
> System.out.println("Queryme :"
> + qname + ","
> + " Result : " + result + " Execution Time in minutes : " +
> TimeInmins);
> }
>   }
> } catch (Exception e) {
> System.out.println("Error reading line: "
> +e.getMessage());
> e.printStackTrace();
>  }
> }
>
> I would like to know if it is ok to use the method controller
> which seems to
> be deprecated now.I must tell you that a xqyery which is
> taking more than 1
> min to execute should have taken less than a minute
> ideally.Thus my total
> execution time has gone up drastically.
>
> I would appreciate any help in this regard.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of
> Michael Kay
> Sent: Thursday, November 17, 2005 8:49 PM
> To: 'Dasgupta, Subhojit (Subhojit)** CTR **'
> Cc: [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> saxon API
>
>
> Could I ask you again - please copy your messages to the list
> so there is an
> archive.
>
> I would suggest you put each of your "queries" into a
> separate function, and
> compile them all as part of a single module. Then have a
> query parameter
> (external global variable) that defines which query you want
> to execute, and
> the main body of your query should test this variable and
> call one of the
> functions as appropriate.
>
>
> declare variable $q external;
> declare function my:q1() {
> ...
> }
> declare function my:q2() {
> ...
> }
> if ($q="q1") then my:q1()
> else if ($q="q2") then my:q2()
> ...
> else error()
>
>
>
> Separate compilation of library modules is something I would like to
> implement at some stage in the future.
>
> Michael Kay
> http://www.saxonica.com/ 
>
> > -----Original Message-----
> > From: Dasgupta, Subhojit (Subhojit)** CTR **
> > [mailto:[hidden email]]
> > Sent: 17 November 2005 14:26
> > To: '[hidden email]'
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with saxon API
> >
> > Hi Michael
> >
> > Thanks a lot  for the valuable inputs.I do agree with you on
> > this point that
> > i have been executing multiple queries as separate
> > expressions within a
> > single query expressed as a list. Ideally i would like to
> > execute individual
> > queries one after another against the same xml data file
> > which is also huge.
> > The issue here is that the xqueries that i have written make
> > use of a common
> > library module(functions defined in the same module) which i
> > imported within
> > the xquery file.The reason for this is that i wanted to
> > import the module
> > once for all the queries in the static context and not load
> > it repeatedly
> > for all the queries.I apologise if i am getting unclear here.
> >
> > I can think of a way whereby for each of the xqueries i would
> > call compile
> > query and then execute them in the dynamic context.But that
> > would also need
> > the external library to be loaded every time. Please correct
> > me if i am
> > wrong.I wanted to avoid this totally. I cant think of a way
> > whereby i can
> > import the library module completely independent of the
> xquery file!.
> >
> > So it would be great if you can suggest me some way of
> importing this
> > library module only once for all the queries.Also i would
> > like to mention
> > that the xml document is being loaded only once using
> buildDocument().
> >
> > I appreciate your effort and valuable time spent.
> >
> > Best Regards
> > Subhojit
> >
> > -----Original Message-----
> > From: [hidden email]
> > [mailto:[hidden email]]On Behalf Of
> > Michael Kay
> > Sent: Wednesday, November 16, 2005 9:37 PM
> > To: [hidden email]
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with
> > saxon API
> >
> >
> > I'm not sure where your difficulty lies. Perhaps it would be
> > easier if you
> > showed your code.
> >
> > Firstly, I don't understand why you want to create multiple
> > queries from the
> > same .xq file. Why not create one query and execute it
> multiple times?
> >
> > Secondly, if you do need to create multiple queries, there is
> > no difficulty
> > doing so. You just call compileQuery() more than once. You
> > can either use
> > the same StaticContext or a different StaticContext for each query.
> >
> > There's also no problem in parsing the source document once
> > and then using
> > it repeatedly. Call the buildDocument() method in
> > StaticQueryContext once to
> > build the DocumentInfo, then supply this DocumentInfo as the
> > context item in
> > the dynamic context for each query evaluation.
> >
> > I hope this helps.
> >
> > Michael Kay
> > http://www.saxonica.com/ 
> >
> > > -----Original Message-----
> > > From: [hidden email]
> > > [mailto:[hidden email]] On Behalf Of
> > > Dasgupta, Subhojit (Subhojit)** CTR **
> > > Sent: 16 November 2005 14:23
> > > To: '[hidden email]'
> > > Subject: [saxon] Query regarding execution of multiple
> > > xqueries with saxon API
> > >
> > > Hi
> > >
> > > I am using Saxon B 8.5.1 version and i need to execute
> > > multiple xqueries in
> > > my application. I have written a java API that compiles the
> > > xqueries that i
> > > have and the library module  in a static context and then
> > > execute those
> > > xqueries in a dynamic context.Also it loads the xml file in
> > > the dynamic
> > > context. For this i used the standard  functions provided by
> > > saxon (like
> > > compileQuery(), evaluateSingle() etc). The issue that i am
> > > facing is that
> > > the above works fine when i am using a single xquery.But in
> > > my application
> > > i need to execute multiple xqueries from the same  .xq file
> > > in the same
> > > context and collect some performance statistics for all those
> > > xqueries (viz
> > > execution time for each xquery etc). Also i cannot afford to
> > > load the XML
> > > file more than once as it as huge!!.
> > >
> > > With the setup i have as mentioned here, i am unable to execute
> > > multiple xqueries. Does Saxon provide an alternate way of
> > > achieving the same
> > > and collect performance data for each and every xquery that
> > i execute?
> > >
> > > I am looking forward to some help in this regard as to how
> > > multiple xqueries
> > > can be created in saxon without creating  a context for every
> > > new query from
> > > the xquery file.
> > >
> > > Best Regards
> > > Subhojit
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by the JBoss Inc.  Get
> > Certified Today
> > > Register for a JBoss Training Course.  Free Certification Exam
> > > for All Training Attendees Through End of 2005. For more
> info visit:
> > > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > > _______________________________________________
> > > saxon-help mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/saxon-help
> > >
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc.  Get
> Certified Today
> > Register for a JBoss Training Course.  Free Certification Exam
> > for All Training Attendees Through End of 2005. For more info visit:
> > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > _______________________________________________
> > saxon-help mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/saxon-help
> >
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
> Register for a JBoss Training Course.  Free Certification Exam
> for All Training Attendees Through End of 2005. For more info visit:
> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep
> through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  
> DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Dasgupta, Subhojit (Subhojit)** CTR **
In reply to this post by Dasgupta, Subhojit (Subhojit)** CTR **
Hi

I wrote the xquery "every $x in X satisfies E"  as "not(exists(X[not(E)]))"
and executed it against the same 90MB file.But the performance seems to be
no good than before.In fact it is taking a while more for execution
now.Could you please suggest some other optimisations?

Thanks
Subhojit

-----Original Message-----
From: Michael Kay [mailto:[hidden email]]
Sent: Friday, December 02, 2005 5:52 AM
To: 'Dasgupta, Subhojit (Subhojit)** CTR **';
[hidden email]
Subject: RE: [saxon] Query regarding execution of multiple xqueries with
s axon API


I don't think there's any problem with your calling Java code, I suspect the
problem is with your query. You could confirm this by running the same query
from the command line.

When investigating this kind of performance problem, it's useful to explore
how the execution time changes as the data file increases in size. In
particular, it's useful to know whether the execution time varies linearly
or quadratically with the data size.

Your query as written looks rather odd. I would like to see the complete
query, that is, including the xpath expression and the functions f1 and f2.
Without the complete query, I can't tell what the optimizer is likely to do
with it. In fact, Saxon doesn't try very hard to optimize "some" and "every"
expressions, because they aren't used all that often; you might find it
helps to rewrite the expression by hand. You can always rewrite

every $x in X satisfies E

as

not(exists(X[not(E)]))

provided you take care to replace references to the $x with references to
the context item.

Michael Kay
http://www.saxonica.com/

> -----Original Message-----
> From: Dasgupta, Subhojit (Subhojit)** CTR **
> [mailto:[hidden email]]
> Sent: 01 December 2005 14:27
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with s axon API
>
> Hi Michael
>
> Just to throw some more light into my problem , below is one
> of the sample
> queries which i am trying to execute:
>
> declare namespace ns = "http://www.xyz.com";
>
> every $x in ($ns:<xpath>) satisfies (every $y in ($x[not((ns:f1() >
> ns:f2()))]) satisfies (false()));
>
> When i executed the above query against a 90MB XML file, it
> took 1.44 mins
> for execution.The loading time for me has been 1 min 7 secs
> consistently for
> the setup i have.
>
> Would it be possible to bring this time to less than a minute or few
> seconds?Will the query plan anlysis do any good here?
>
> Kindly help.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of Dasgupta,
> Subhojit (Subhojit)** CTR **
> Sent: Thursday, December 01, 2005 6:37 PM
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> s axon API
>
>
> Hi Michael
>
> I have developed a Java API that internally uses saxon 8.5.1
> methods to
> execute xqueries from an external file.As you had suggested
> earlier , i am
> now loading the big XML file (~90MB) using buildDocument()
> only once.Also
> all my xqueries are defined as functions in the query.xq
> file.Which query to
> execute is being controlled from the Java API. While
> executing the xqueries
> i found that the execution time is much more than what i had
> expected for
> every xquery in my file.Is it possible that for each of these
> queries the
> xml document is getting loaded every time inadvertently?
>
> My pseudo code is as follows:
>
>         final Configuration config = new Configuration();
>         final StaticQueryContext sqc = new StaticQueryContext(config);
>
> final XQueryExpression exp = sqc.compileQuery(new
> FileReader("./query.xq"));
>       DynamicQueryContext dynamicContext = new
> DynamicQueryContext(config);
>
> dynamicContext.setContextNode(sqc.buildDocument(new
> StreamSource("./Mydata.xml")));
>
> String s;
>         BufferedReader nr = new BufferedReader(new
> FileReader("querynames.xml"));
>   try {
> while ( (s = nr.readLine()) != null ) {
> String sp = "<qnames>";
> String ep = "</qnames>";
> if (s.startsWith(sp)) {
>     String str = s.substring(6);
>     String qname = str.replaceAll(ep, "");
> final UserFunction fn =
> exp.getStaticContext().getUserDefinedFunction("http://www.myfu
> nc.com/query",
> "QueryMain", 1);
> final Value[] arglist = new Value[1];
>      arglist[0] = new StringValue(qname);
>         final Controller controller =
> exp.getController();
> startTime = System.currentTimeMillis();
>             final ValueRepresentation result =
> fn.call(arglist,
> controller);
> endTime = System.currentTimeMillis();
>
> time = (endTime - startTime)/1000.0;
> TimeInmins = time/60.0;
>
> System.out.println("Queryme :"
> + qname + ","
> + " Result : " + result + " Execution Time in minutes : " +
> TimeInmins);
> }
>   }
> } catch (Exception e) {
> System.out.println("Error reading line: "
> +e.getMessage());
> e.printStackTrace();
>  }
> }
>
> I would like to know if it is ok to use the method controller
> which seems to
> be deprecated now.I must tell you that a xqyery which is
> taking more than 1
> min to execute should have taken less than a minute
> ideally.Thus my total
> execution time has gone up drastically.
>
> I would appreciate any help in this regard.
>
> Thanks
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of
> Michael Kay
> Sent: Thursday, November 17, 2005 8:49 PM
> To: 'Dasgupta, Subhojit (Subhojit)** CTR **'
> Cc: [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> saxon API
>
>
> Could I ask you again - please copy your messages to the list
> so there is an
> archive.
>
> I would suggest you put each of your "queries" into a
> separate function, and
> compile them all as part of a single module. Then have a
> query parameter
> (external global variable) that defines which query you want
> to execute, and
> the main body of your query should test this variable and
> call one of the
> functions as appropriate.
>
>
> declare variable $q external;
> declare function my:q1() {
> ...
> }
> declare function my:q2() {
> ...
> }
> if ($q="q1") then my:q1()
> else if ($q="q2") then my:q2()
> ...
> else error()
>
>
>
> Separate compilation of library modules is something I would like to
> implement at some stage in the future.
>
> Michael Kay
> http://www.saxonica.com/ 
>
> > -----Original Message-----
> > From: Dasgupta, Subhojit (Subhojit)** CTR **
> > [mailto:[hidden email]]
> > Sent: 17 November 2005 14:26
> > To: '[hidden email]'
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with saxon API
> >
> > Hi Michael
> >
> > Thanks a lot  for the valuable inputs.I do agree with you on
> > this point that
> > i have been executing multiple queries as separate
> > expressions within a
> > single query expressed as a list. Ideally i would like to
> > execute individual
> > queries one after another against the same xml data file
> > which is also huge.
> > The issue here is that the xqueries that i have written make
> > use of a common
> > library module(functions defined in the same module) which i
> > imported within
> > the xquery file.The reason for this is that i wanted to
> > import the module
> > once for all the queries in the static context and not load
> > it repeatedly
> > for all the queries.I apologise if i am getting unclear here.
> >
> > I can think of a way whereby for each of the xqueries i would
> > call compile
> > query and then execute them in the dynamic context.But that
> > would also need
> > the external library to be loaded every time. Please correct
> > me if i am
> > wrong.I wanted to avoid this totally. I cant think of a way
> > whereby i can
> > import the library module completely independent of the
> xquery file!.
> >
> > So it would be great if you can suggest me some way of
> importing this
> > library module only once for all the queries.Also i would
> > like to mention
> > that the xml document is being loaded only once using
> buildDocument().
> >
> > I appreciate your effort and valuable time spent.
> >
> > Best Regards
> > Subhojit
> >
> > -----Original Message-----
> > From: [hidden email]
> > [mailto:[hidden email]]On Behalf Of
> > Michael Kay
> > Sent: Wednesday, November 16, 2005 9:37 PM
> > To: [hidden email]
> > Subject: RE: [saxon] Query regarding execution of multiple
> > xqueries with
> > saxon API
> >
> >
> > I'm not sure where your difficulty lies. Perhaps it would be
> > easier if you
> > showed your code.
> >
> > Firstly, I don't understand why you want to create multiple
> > queries from the
> > same .xq file. Why not create one query and execute it
> multiple times?
> >
> > Secondly, if you do need to create multiple queries, there is
> > no difficulty
> > doing so. You just call compileQuery() more than once. You
> > can either use
> > the same StaticContext or a different StaticContext for each query.
> >
> > There's also no problem in parsing the source document once
> > and then using
> > it repeatedly. Call the buildDocument() method in
> > StaticQueryContext once to
> > build the DocumentInfo, then supply this DocumentInfo as the
> > context item in
> > the dynamic context for each query evaluation.
> >
> > I hope this helps.
> >
> > Michael Kay
> > http://www.saxonica.com/ 
> >
> > > -----Original Message-----
> > > From: [hidden email]
> > > [mailto:[hidden email]] On Behalf Of
> > > Dasgupta, Subhojit (Subhojit)** CTR **
> > > Sent: 16 November 2005 14:23
> > > To: '[hidden email]'
> > > Subject: [saxon] Query regarding execution of multiple
> > > xqueries with saxon API
> > >
> > > Hi
> > >
> > > I am using Saxon B 8.5.1 version and i need to execute
> > > multiple xqueries in
> > > my application. I have written a java API that compiles the
> > > xqueries that i
> > > have and the library module  in a static context and then
> > > execute those
> > > xqueries in a dynamic context.Also it loads the xml file in
> > > the dynamic
> > > context. For this i used the standard  functions provided by
> > > saxon (like
> > > compileQuery(), evaluateSingle() etc). The issue that i am
> > > facing is that
> > > the above works fine when i am using a single xquery.But in
> > > my application
> > > i need to execute multiple xqueries from the same  .xq file
> > > in the same
> > > context and collect some performance statistics for all those
> > > xqueries (viz
> > > execution time for each xquery etc). Also i cannot afford to
> > > load the XML
> > > file more than once as it as huge!!.
> > >
> > > With the setup i have as mentioned here, i am unable to execute
> > > multiple xqueries. Does Saxon provide an alternate way of
> > > achieving the same
> > > and collect performance data for each and every xquery that
> > i execute?
> > >
> > > I am looking forward to some help in this regard as to how
> > > multiple xqueries
> > > can be created in saxon without creating  a context for every
> > > new query from
> > > the xquery file.
> > >
> > > Best Regards
> > > Subhojit
> > >
> > >
> > >
> > > -------------------------------------------------------
> > > This SF.Net email is sponsored by the JBoss Inc.  Get
> > Certified Today
> > > Register for a JBoss Training Course.  Free Certification Exam
> > > for All Training Attendees Through End of 2005. For more
> info visit:
> > > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > > _______________________________________________
> > > saxon-help mailing list
> > > [hidden email]
> > > https://lists.sourceforge.net/lists/listinfo/saxon-help
> > >
> >
> >
> >
> >
> > -------------------------------------------------------
> > This SF.Net email is sponsored by the JBoss Inc.  Get
> Certified Today
> > Register for a JBoss Training Course.  Free Certification Exam
> > for All Training Attendees Through End of 2005. For more info visit:
> > http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> > _______________________________________________
> > saxon-help mailing list
> > [hidden email]
> > https://lists.sourceforge.net/lists/listinfo/saxon-help
> >
>
>
>
>
> -------------------------------------------------------
> This SF.Net email is sponsored by the JBoss Inc.  Get Certified Today
> Register for a JBoss Training Course.  Free Certification Exam
> for All Training Attendees Through End of 2005. For more info visit:
> http://ads.osdn.com/?ad_id=7628&alloc_id=16845&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep
> through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  
> DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>



-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Michael Kay
In reply to this post by Dasgupta, Subhojit (Subhojit)** CTR **
>
> Some interesting findings from my side.I executed the same
> query i mentioned
> earlier from command line now with the java -server option
> and i am able to
> get some optimisation.The query finished execution in less than 20
> secs.However when executed using the API it is taking now 1 min 20 sec
> (reduction in 20 secs .It was 1.44 before).Also i have
> observed that the
> loading time has come down after i have used this -server option .
>
> But still i am not able to get the optimisation in the API
> way as i got in
> the command line.Does it mean that the
> command line execution is more efficient in terms of performance
> optimisation? Is it possible that the xml document is being
> loaded every
> time per query in the API scheme of things?
>

You still haven't shown me the actual query you are executing, or given any
information about the source document other than its size, which makes it
difficult for me to give you much help.

The API you are using in your Java application is not recommended for
general use, because it doesn't do thorough parameter validation. It was
designed for a use case involving many thousands of very simple queries,
where it was important to minimize the start-up cost. These arguments don't
apply to your case; but I can't see any reason why using this Java API
should ever be slower than executing the query from the command line. The
problem must lie in something that you haven't shared with me.

It would be helpful if you can send me the material (including a cut-down
data file) in a form that I can experiment with: otherwise, I can only
speculate.

Michael Kay
http://www.saxonica.com/




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Dasgupta, Subhojit (Subhojit)** CTR **
In reply to this post by Dasgupta, Subhojit (Subhojit)** CTR **
Hi Michael

Please find attached a xquery file with the sample xml data file which i am
using in my application .The xml file has just the kind of elements that the
xquery operates on.The actual file that i use here has a size of 90MB.Since
these are already in production, i am not supposed to share the data to
outside world.So please feel free to make use of the samples provided
herewith and let me know if that is what you were looking for or you still
want some more info from my side.

Is it possible to have some optimizations in the java API itself?


Best Regards
Subhojit



-----Original Message-----
From: [hidden email]
[mailto:[hidden email]]On Behalf Of Michael Kay
Sent: Friday, December 02, 2005 9:03 PM
To: 'Dasgupta, Subhojit (Subhojit)** CTR **';
[hidden email]
Subject: RE: [saxon] Query regarding execution of multiple xqueries with
s axon API


>
> Some interesting findings from my side.I executed the same
> query i mentioned
> earlier from command line now with the java -server option
> and i am able to
> get some optimisation.The query finished execution in less than 20
> secs.However when executed using the API it is taking now 1 min 20 sec
> (reduction in 20 secs .It was 1.44 before).Also i have
> observed that the
> loading time has come down after i have used this -server option .
>
> But still i am not able to get the optimisation in the API
> way as i got in
> the command line.Does it mean that the
> command line execution is more efficient in terms of performance
> optimisation? Is it possible that the xml document is being
> loaded every
> time per query in the API scheme of things?
>
You still haven't shown me the actual query you are executing, or given any
information about the source document other than its size, which makes it
difficult for me to give you much help.

The API you are using in your Java application is not recommended for
general use, because it doesn't do thorough parameter validation. It was
designed for a use case involving many thousands of very simple queries,
where it was important to minimize the start-up cost. These arguments don't
apply to your case; but I can't see any reason why using this Java API
should ever be slower than executing the query from the command line. The
problem must lie in something that you haven't shared with me.

It would be helpful if you can send me the material (including a cut-down
data file) in a form that I can experiment with: otherwise, I can only
speculate.

Michael Kay
http://www.saxonica.com/




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help


sample.xml (347 bytes) Download Attachment
sample.xq (600 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with s axon API

Michael Kay
Some preliminary results on this:

I've created a test data generator in XSLT to produce sample datasets of
various sizes, and have run the query from the command line against these.
This is showing linear performance for data sizes up to around 3200K records
(36Mb). Above that size the performance becomes non-linear: it seems that
the algorithm is therefore linear in time but not in space requirements. The
variations that you are seeing in performance under different circumstances
are probably due, I strongly suspect, to differences in the availability of
Java heap memory in the different environments.

However, I'm currently measuring tree construction and query execution
combined. It's possible that the query execution time is non-linear and this
is masked at smaller data sizes by the dominant cost of the tree
construction.

Incidentally both your source document and query used namespaces that
weren't declared. I replaced your call to the undeclared function
ns:convert-to-int() by a simple call on xs:int().

Investigations continue...

Michael Kay

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of
> Dasgupta, Subhojit (Subhojit)** CTR **
> Sent: 06 December 2005 11:16
> To: '[hidden email]'
> Cc: '[hidden email]'
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with s axon API
>
> Hi Michael
>
> Please find attached a xquery file with the sample xml data
> file which i am
> using in my application .The xml file has just the kind of
> elements that the
> xquery operates on.The actual file that i use here has a size
> of 90MB.Since
> these are already in production, i am not supposed to share
> the data to
> outside world.So please feel free to make use of the samples provided
> herewith and let me know if that is what you were looking for
> or you still
> want some more info from my side.
>
> Is it possible to have some optimizations in the java API itself?
>
>
> Best Regards
> Subhojit
>
>
>
> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]]On Behalf Of
> Michael Kay
> Sent: Friday, December 02, 2005 9:03 PM
> To: 'Dasgupta, Subhojit (Subhojit)** CTR **';
> [hidden email]
> Subject: RE: [saxon] Query regarding execution of multiple
> xqueries with
> s axon API
>
>
> >
> > Some interesting findings from my side.I executed the same
> > query i mentioned
> > earlier from command line now with the java -server option
> > and i am able to
> > get some optimisation.The query finished execution in less than 20
> > secs.However when executed using the API it is taking now 1
> min 20 sec
> > (reduction in 20 secs .It was 1.44 before).Also i have
> > observed that the
> > loading time has come down after i have used this -server option .
> >
> > But still i am not able to get the optimisation in the API
> > way as i got in
> > the command line.Does it mean that the
> > command line execution is more efficient in terms of performance
> > optimisation? Is it possible that the xml document is being
> > loaded every
> > time per query in the API scheme of things?
> >
>
> You still haven't shown me the actual query you are
> executing, or given any
> information about the source document other than its size,
> which makes it
> difficult for me to give you much help.
>
> The API you are using in your Java application is not recommended for
> general use, because it doesn't do thorough parameter
> validation. It was
> designed for a use case involving many thousands of very
> simple queries,
> where it was important to minimize the start-up cost. These
> arguments don't
> apply to your case; but I can't see any reason why using this Java API
> should ever be slower than executing the query from the
> command line. The
> problem must lie in something that you haven't shared with me.
>
> It would be helpful if you can send me the material
> (including a cut-down
> data file) in a form that I can experiment with: otherwise, I can only
> speculate.
>
> Michael Kay
> http://www.saxonica.com/
>
>
>
>
> -------------------------------------------------------
> This SF.net email is sponsored by: Splunk Inc. Do you grep
> through log files
> for problems?  Stop!  Download the new AJAX search engine that makes
> searching your log files as easy as surfing the  web.  
> DOWNLOAD SPLUNK!
> http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
> _______________________________________________
> saxon-help mailing list
> [hidden email]
> https://lists.sourceforge.net/lists/listinfo/saxon-help
>
>




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

RE: Query regarding execution of multiple xqueries with saxon API

Michael Kay
I think that these oddities in measured performance come down to the fact
that you are executing a very peculiar query:

every $x in /TOP/TEST satisfies
 every $y in $x[@A > @B] satisfies false()

If this is executed in pipelined mode, then the query will return false as
soon as it encouters the first TEST element that has @A > @B. I generated
data at random, so such an element will almost invariably be found in the
first three TEST elements or so. This results in the query finishing (and
returning false) within about 1.5ms, regardless of the data size. (The tree
building, of course, takes much longer than this, and is proportional to
size until you start getting short of memory, at which stage the time grows
exponentially.)

I haven't been able to reproduce the differences you observed between
different ways of running the query.

It might be that my data wasn't typical of the data you were using.

I would suggest that if you want to make further performance measurements,
you try a query that's more representative of real life. You're going to
have to send me a test that I can reproduce - so long as I'm doing something
different from what you're doing, the results aren't going to be very
useful.

Michael Kay
http://www.saxonica.com/
 




-------------------------------------------------------
This SF.net email is sponsored by: Splunk Inc. Do you grep through log files
for problems?  Stop!  Download the new AJAX search engine that makes
searching your log files as easy as surfing the  web.  DOWNLOAD SPLUNK!
http://ads.osdn.com/?ad_id=7637&alloc_id=16865&op=click
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help