Large Size XML files for data transfer

[RAJ]

Hi,
Need some advice with regards to XML files. Here is what I am trying to
achieve:

I have an SQL Server 7.0 database and a Web service running on a Windows
2003 enterprise machine (for development). The web service (developed using
VB.NET) creates an XML file which contains most of the database table. The
creation of this file takes less then couple of minutes. This XML is about
57 MB in size and is used to build a local database. The file is then zipped
to about 2.5 MB and sent to the calling application.

On the client the file is unzipped to 57 MB file, before the application
starts reading it. I am using the load method of the DOM object model to
check whether it is a well formed XML. The problem starts as soon as I try
to load this XML file. The client machine becomes unresponsive for a long
time and the whole process from there on is very slow as I assume that the
XML is loaded into memory while it is parsed. I have to read the whole XML
file and do not need to search for any nodes/tags inside the XML.

I would like to know whether it is right to use XML in this scenario? Am I
better off by generating a simple delimited file instead of an XML file? The
size of the XML file is at least 3 folds as compared to any other delimited
file.

Any advice in this regard will be helpful.

Thanks and regards,

RAJ

[Kondratyev Denis]

"RAJ" <anonymous@discussions.microsoft.com> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ
ÓÌÅÄÕÀÝÅÅ: news:#HTqv9OZEHA.3132@TK2MSFTNGP10.phx.gbl...

> Hi,
> Need some advice with regards to XML files. Here is what I am trying to
> achieve:
>
> I have an SQL Server 7.0 database and a Web service running on a Windows
> 2003 enterprise machine (for development). The web service (developed

using

> VB.NET) creates an XML file which contains most of the database table. The
> creation of this file takes less then couple of minutes. This XML is about
> 57 MB in size and is used to build a local database. The file is then

zipped

> to about 2.5 MB and sent to the calling application.
>
> On the client the file is unzipped to 57 MB file, before the application
> starts reading it. I am using the load method of the DOM object model to
> check whether it is a well formed XML. The problem starts as soon as I try
> to load this XML file. The client machine becomes unresponsive for a long
> time and the whole process from there on is very slow as I assume that the
> XML is loaded into memory while it is parsed. I have to read the whole XML
> file and do not need to search for any nodes/tags inside the XML.

Use asynhrone calls of web service methods for avoid unresponsive
application.

> I would like to know whether it is right to use XML in this scenario? Am I
> better off by generating a simple delimited file instead of an XML file?

The

> size of the XML file is at least 3 folds as compared to any other

delimited

> file.

If u store some table with different structure imho xml file is better
solution.

[Iain]

Personally, if you are sending a single table, I would send a flat file of
some kind - CSV is fine.

However, if you want to use XML then I would use an XMLReader rather than
an XMLDocument. This should be dramatically faster (I'm presuming that the
ultimate database target is NOT an XMLDocument).

I would be tempted to including the unzipping as part of the reading
processing by using zlib or something. However, a quick glance at the docs
suggest this will not be straightforward.

Iain

"RAJ" <anonymous@discussions.microsoft.com> wrote in message
news:#HTqv9OZEHA.3132@TK2MSFTNGP10.phx.gbl...

> Hi,
> Need some advice with regards to XML files. Here is what I am trying to
> achieve:
>
> I have an SQL Server 7.0 database and a Web service running on a Windows
> 2003 enterprise machine (for development). The web service (developed

using

> VB.NET) creates an XML file which contains most of the database table. The
> creation of this file takes less then couple of minutes. This XML is about
> 57 MB in size and is used to build a local database. The file is then

zipped

> to about 2.5 MB and sent to the calling application.
>
> On the client the file is unzipped to 57 MB file, before the application
> starts reading it. I am using the load method of the DOM object model to
> check whether it is a well formed XML. The problem starts as soon as I try
> to load this XML file. The client machine becomes unresponsive for a long
> time and the whole process from there on is very slow as I assume that the
> XML is loaded into memory while it is parsed. I have to read the whole XML
> file and do not need to search for any nodes/tags inside the XML.
>
> I would like to know whether it is right to use XML in this scenario? Am I
> better off by generating a simple delimited file instead of an XML file?

The

> size of the XML file is at least 3 folds as compared to any other

delimited

> file.
>
> Any advice in this regard will be helpful.
>
> Thanks and regards,
>
> RAJ
>
>

[RAJ]

Thanks for your reply.
I am sending a part of the database with a few tables. The advantage of
using XML is that it is readable. However, I can change it to use either
tab-separated or CSV format.

The webservice is called asynchronously with the implementation of Zipping
and unzipping using third party tool (AbaleZip) at both the webservice and
client side respectively.

The problem occurs when I use the load command :
dim objXML as XMLDocument
objxml.load(<XML file name>)

The application hangs when reading the file on the client machine.
During development however, the load command is successful but then windows
is out of virtual memory and the reading of the XML file from there on is
very slow!

Never used XMLReader before. But it looks like that I have to use it as it
is a forward only reader which is what I am looking for as I need to read
whole data sequentially. Will I be able to use it without loading the XML?

I have already developed both client and web service part and now in the
process of testing on deployment machine.

If there are any examples of using XMLReader in VB..NET, it would be helpful

Thanks for your help.

Regards,

RAJ

"Iain" <idownsXXXX@dircon.co.uk> wrote in message
news:eNI10SQZEHA.136@TK2MSFTNGP11.phx.gbl...

> Personally, if you are sending a single table, I would send a flat file of
> some kind - CSV is fine.
>
> However, if you want to use XML then I would use an XMLReader rather than
> an XMLDocument. This should be dramatically faster (I'm presuming that

the

> ultimate database target is NOT an XMLDocument).
>
> I would be tempted to including the unzipping as part of the reading
> processing by using zlib or something. However, a quick glance at the

docs

> suggest this will not be straightforward.
>
> Iain
>
> "RAJ" <anonymous@discussions.microsoft.com> wrote in message
> news:#HTqv9OZEHA.3132@TK2MSFTNGP10.phx.gbl...

> > Hi,
> > Need some advice with regards to XML files. Here is what I am trying to
> > achieve:
> >
> > I have an SQL Server 7.0 database and a Web service running on a Windows
> > 2003 enterprise machine (for development). The web service (developed

> using

> > VB.NET) creates an XML file which contains most of the database table.

The

> > creation of this file takes less then couple of minutes. This XML is

about

> > 57 MB in size and is used to build a local database. The file is then

> zipped

> > to about 2.5 MB and sent to the calling application.
> >
> > On the client the file is unzipped to 57 MB file, before the application
> > starts reading it. I am using the load method of the DOM object model to
> > check whether it is a well formed XML. The problem starts as soon as I

try

> > to load this XML file. The client machine becomes unresponsive for a

long

> > time and the whole process from there on is very slow as I assume that

the

> > XML is loaded into memory while it is parsed. I have to read the whole

XML

> > file and do not need to search for any nodes/tags inside the XML.
> >
> > I would like to know whether it is right to use XML in this scenario? Am

I

> > better off by generating a simple delimited file instead of an XML file?

> The

> > size of the XML file is at least 3 folds as compared to any other

> delimited

> > file.
> >
> > Any advice in this regard will be helpful.
> >
> > Thanks and regards,
> >
> > RAJ
> >
> >

>
>

[Kondratyev Denis]

Imho Iain mean XmlTextReader class, becauce XmlReader is abstract class and
u cant use it for work with XML (rtfm). U can find examples in .NET
Framework SDK,
[url]http://msdn.microsoft.com/vcsharp/downloads/samples/23samples/default.aspx[/url] ,
[url]http://support.microsoft.com/default.aspx?scid=kb;en-us;301225[/url] or in Google.

"RAJ" <anonymous@discussions.microsoft.com> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ
ÓÌÅÄÕÀÝÅÅ: news:eyOAC3QZEHA.1152@TK2MSFTNGP09.phx.gbl...

> Thanks for your reply.
> I am sending a part of the database with a few tables. The advantage of
> using XML is that it is readable. However, I can change it to use either
> tab-separated or CSV format.
>
> The webservice is called asynchronously with the implementation of Zipping
> and unzipping using third party tool (AbaleZip) at both the webservice and
> client side respectively.
>
> The problem occurs when I use the load command :
> dim objXML as XMLDocument
> objxml.load(<XML file name>)
>
> The application hangs when reading the file on the client machine.
> During development however, the load command is successful but then

windows

> is out of virtual memory and the reading of the XML file from there on is
> very slow!
>
> Never used XMLReader before. But it looks like that I have to use it as

it

> is a forward only reader which is what I am looking for as I need to read
> whole data sequentially. Will I be able to use it without loading the XML?
>
> I have already developed both client and web service part and now in the
> process of testing on deployment machine.
>
> If there are any examples of using XMLReader in VB..NET, it would be

helpful

>
> Thanks for your help.
>
> Regards,
>
> RAJ

[RAJ]

Thanks a lot. You help is much appreciated.

regards,
RAJ

"Kondratyev Denis" <info@visualdesign.ru> wrote in message
news:cck9je$jq3$1@news.rol.ru...

> Imho Iain mean XmlTextReader class, becauce XmlReader is abstract class

and

> u cant use it for work with XML (rtfm). U can find examples in .NET
> Framework SDK,
> [url]http://msdn.microsoft.com/vcsharp/downloads/samples/23samples/default.aspx[/url]

,

> [url]http://support.microsoft.com/default.aspx?scid=kb;en-us;301225[/url] or in

Google.

>
> "RAJ" <anonymous@discussions.microsoft.com> ÓÏÏÂÝÉÌ/ÓÏÏÂÝÉÌÁ × ÎÏ×ÏÓÔÑÈ
> ÓÌÅÄÕÀÝÅÅ: news:eyOAC3QZEHA.1152@TK2MSFTNGP09.phx.gbl...

> > Thanks for your reply.
> > I am sending a part of the database with a few tables. The advantage of
> > using XML is that it is readable. However, I can change it to use

either

> > tab-separated or CSV format.
> >
> > The webservice is called asynchronously with the implementation of

Zipping

> > and unzipping using third party tool (AbaleZip) at both the webservice

and

> > client side respectively.
> >
> > The problem occurs when I use the load command :
> > dim objXML as XMLDocument
> > objxml.load(<XML file name>)
> >
> > The application hangs when reading the file on the client machine.
> > During development however, the load command is successful but then

> windows

> > is out of virtual memory and the reading of the XML file from there on

is

> > very slow!
> >
> > Never used XMLReader before. But it looks like that I have to use it as

> it

> > is a forward only reader which is what I am looking for as I need to

read

> > whole data sequentially. Will I be able to use it without loading the

XML?

> >
> > I have already developed both client and web service part and now in the
> > process of testing on deployment machine.
> >
> > If there are any examples of using XMLReader in VB..NET, it would be

> helpful

> >
> > Thanks for your help.
> >
> > Regards,
> >
> > RAJ

>
>

[Iain]

"Kondratyev Denis" <info@visualdesign.ru> wrote in message
news:cck9je$jq3$1@news.rol.ru...

> Imho Iain mean XmlTextReader class, becauce XmlReader is abstract class

and

> u cant use it for work with XML (rtfm). U can find examples in .NET
> Framework SDK,
> [url]http://msdn.microsoft.com/vcsharp/downloads/samples/23samples/default.aspx[/url]

,

> [url]http://support.microsoft.com/default.aspx?scid=kb;en-us;301225[/url] or in

Google.

>

Sorry. Yes. - my earlier point was it should be possible to write an
XmlZippedTextReader class (and it's counterpart) which would be useful..

However, I'm not volounteering.

Iain