java -d64 impact on saxon processing

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

java -d64 impact on saxon processing

Peter.Rushforth
Hi,

I have discovered that one can't allocate more than -Xmx4g with the default
java invocation on Solaris 8, but that if one uses the -d64 option it is possible to
exceed the 4g barrier.

I was wondering if anyone has any experience with the impact of using this option
and large quantities of heap space on saxon processing?

We need to process an xml file of approximately 1.2 Gb, and apparently need to either
adopt another strategy or allocate heaps of heap :).

Cheers,

Peter
N�HS^�隊X���'���u��<�ڂ�.���y�"� �*m�x%jx.j���^�קvƩ�X�jب�ȧ��m�ݚ���v&��קv�^�+����j�Z���{az���^��h��஋�n���)��{h�����ا�׫�+h�(m�����Z��jY�w��ǥrg�y$��Oxḝۍ{�GZ�׮6��h���f��)��+-��h���X���(��~��zw���i����l���q���z���l�X��)ߣ�Ɖ�zZ
Reply | Threaded
Open this post in threaded view
|

Re: java -d64 impact on saxon processing

Peter.Rushforth
Here goes with a resend - hopefully this can be seen by all:

Hi,

I have discovered that one can't allocate more than -Xmx4g with the default java invocation on Solaris 8, but that if one uses the -d64 option it is possible to
exceed the 4g barrier.

I was wondering if anyone has any experience with the impact of using this option and large quantities of heap space on saxon processing?

We need to process an xml file of approximately 1.2 Gb, and apparently need to either adopt another strategy or allocate heaps of heap :).

Cheers,

Peter

 
 <<Peter Rushforth ([hidden email]).vcf>>

Peter Rushforth (Peter.Rushforth@statcan.ca).vcf (712 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: Re: java -d64 impact on saxon processing

Michael Kay
It's certainly beyond my experience, so I'll be interested to see how you
get on.

There's a change in 8.7 which should help: it no longer requires that all
the storage for the text of a document should be allocated in one contiguous
array. The changes that cause whitespace text nodes to be compressed might
also help, though usually with a document of this size I would expect you to
be stripping whitespace anyway.

However, if you were asking me to help with this problem, the first thing I
would look for is a way of breaking it down into smaller parts.

Michael Kay
http://www.saxonica.com/
 

> -----Original Message-----
> From: [hidden email]
> [mailto:[hidden email]] On Behalf Of
> [hidden email]
> Sent: 27 February 2006 17:20
> To: [hidden email]
> Subject: [saxon] Re: java -d64 impact on saxon processing
>
> Here goes with a resend - hopefully this can be seen by all:
>
> Hi,
>
> I have discovered that one can't allocate more than -Xmx4g
> with the default java invocation on Solaris 8, but that if
> one uses the -d64 option it is possible to
> exceed the 4g barrier.
>
> I was wondering if anyone has any experience with the impact
> of using this option and large quantities of heap space on
> saxon processing?
>
> We need to process an xml file of approximately 1.2 Gb, and
> apparently need to either adopt another strategy or allocate
> heaps of heap :).
>
> Cheers,
>
> Peter
>
>  
>  <<Peter Rushforth ([hidden email]).vcf>>
>




-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

Re: Re: java -d64 impact on saxon processing

Nigel Whitaker
In reply to this post by Peter.Rushforth

On 27 Feb 2006, at 17:19, <[hidden email]> wrote:

> Here goes with a resend - hopefully this can be seen by all:
>
> Hi,
>
> I have discovered that one can't allocate more than -Xmx4g with the  
> default java invocation on Solaris 8, but that if one uses the -d64  
> option it is possible to
> exceed the 4g barrier.

Indeed,  Solaris' 32 bit JVMs are fairly unusual in getting close to  
the 4GB
barrier, most other 32 bit JVMs (MacOSX, Windows, Linux) will only  
get to
around 2GB because of the way the 32 bit address space is divided  
between
'user space' and 'kernel space'.

With Solaris 9 I found that:  -XX:+UseMPSS had a beneficial effect
on performance - perhaps 10% improvement in runtime with big data

I think -d64 implies -server on recent JVMs, but if not,
also try adding -server


>
> I was wondering if anyone has any experience with the impact of  
> using this option
> and large quantities of heap space on saxon processing?

I've done big processing jobs with command lines like:

   java -d64 -Xmx12g ....

This was a pipeline including Saxon XSLFilters/TransformerHandlers,
our comparator and Java XSLFilter code.  It was certainly possible
process XML input files around 1GB in size.

>
> We need to process an xml file of approximately 1.2 Gb, and apparently
> need to either adopt another strategy or allocate heaps of heap :).

If you've got the RAM/money its certainly possible!  There are servers
available with hundreds of GB, perhaps 1TB, of RAM, most of which can
be used as heap space for a single JVM instance.


Cheers,

Nigel


--
Nigel Whitaker,  DeltaXML: "Change control for XML, in XML"
[hidden email]    http://www.deltaxml.com
Free Online XML Comparison service:  http://compare.deltaxml.com




-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
http://sel.as-us.falkag.net/sel?cmd=lnk&kid=110944&bid=241720&dat=121642
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help
Reply | Threaded
Open this post in threaded view
|

Re: Re: java -d64 impact on saxon processing

Andrew Welch
On 2/28/06, Nigel Whitaker <[hidden email]> wrote:

>
> On 27 Feb 2006, at 17:19, <[hidden email]> wrote:
>
> > Here goes with a resend - hopefully this can be seen by all:
> >
> > Hi,
> >
> > I have discovered that one can't allocate more than -Xmx4g with the
> > default java invocation on Solaris 8, but that if one uses the -d64
> > option it is possible to
> > exceed the 4g barrier.
>
> Indeed,  Solaris' 32 bit JVMs are fairly unusual in getting close to
> the 4GB
> barrier, most other 32 bit JVMs (MacOSX, Windows, Linux) will only
> get to
> around 2GB because of the way the 32 bit address space is divided
> between
> 'user space' and 'kernel space'.
>
> With Solaris 9 I found that:  -XX:+UseMPSS had a beneficial effect
> on performance - perhaps 10% improvement in runtime with big data
>
> I think -d64 implies -server on recent JVMs, but if not,
> also try adding -server
>
>
> >
> > I was wondering if anyone has any experience with the impact of
> > using this option
> > and large quantities of heap space on saxon processing?
>
> I've done big processing jobs with command lines like:
>
>    java -d64 -Xmx12g ....
>
> This was a pipeline including Saxon XSLFilters/TransformerHandlers,
> our comparator and Java XSLFilter code.  It was certainly possible
> process XML input files around 1GB in size.
>
> >
> > We need to process an xml file of approximately 1.2 Gb, and apparently
> > need to either adopt another strategy or allocate heaps of heap :).
>
> If you've got the RAM/money its certainly possible!  There are servers
> available with hundreds of GB, perhaps 1TB, of RAM, most of which can
> be used as heap space for a single JVM instance.

This thread might be of help - if the OP was getting an OutOfMemoryError:

http://forum.java.sun.com/thread.jspa?threadID=707882


-------------------------------------------------------
This SF.Net email is sponsored by xPML, a groundbreaking scripting language
that extends applications into web and mobile media. Attend the live webcast
and join the prime developer group breaking into this new coding territory!
<a href="http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642">http://sel.as-us.falkag.net/sel?cmd=lnk&kid0944&bid$1720&dat1642
_______________________________________________
saxon-help mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/saxon-help