 |
C++Talk.NET
C++ language newsgroups
|
| View previous topic :: View next topic |
| Author |
Message |
SIgnOff
Guest
|
 Posted: Thu Mar 17, 2005 9:55 pm Post subject: What the fastest way to copy strings from file to a vector i |
 |
|
What the fastest way to copy strings from file to a vector in STL?
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
 |
rajkumar@hotmail.com
Guest
|
| Posted: Fri Mar 18, 2005 1:18 pm Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
its all depends on what you mean by fastest...............you can read
it string by string and push_back into a stl vector............but u
might better results if you write to the file the size of the vector
and the size of individual strings... this way you can reserve the
corresponding sizes before you read them for better performance
Raj
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
dietmar_kuehl@yahoo.com
Guest
|
| Posted: Fri Mar 18, 2005 1:19 pm Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
SIgnOff wrote:
| Quote: | What the fastest way to copy strings from file to a vector in STL?
|
You would need to measure the various approaches. Here are some
proposals:
- The most idiomatic approach which can be made the fastest
approach but probably isn't for most implementations is this
(all the code excerpts lack error checking and narrow
characters for brevity):
std::ifstream in("some.file", std::ios_base::binary);
std::string str((std::istreambuf_iterator<char>(in)),
std::istreambuf_iterator<char>());
Theoretically, this code could mmap(2) (or platform specific
equivalent) the file, allocate a suitable chunk of memory, and
copy it there. Even if mmap(2)ing is no option, this code could
take advantage of the segmented sequence optimization and still
do a reasonable job although this would probably require several
copies of the file's data to grow the string's internal
representation.
- Writing stream buffers to stream buffers is likely to be most
efficient approach with typical standard C++ library
implementations using IOStreams:
std::ifstream in("some.file", std::ios_base::binary);
std::ostringstream out;
out << in.rdbuf();
std::string str;
out.str().swap(str);
- If neither of these yields reasonable performance, you might
try a loop constructing the string yourself using either C++'
IOStreams or C's stdio. I would guess that using C's stdio
actually gives the best performance with typical implementations
altlhough this is not necessarily so.
- If this still does not yield sufficient performance, you are up
to platform specific approaches like mmap(2)ing (or equivalent)
the file and possibly just using the corresponding memory section
rather than copying the memory to a 'std::string'.
Similar questions have come up in the past and I think that Alex
Vinokur has posted speed comparisons on this in the past including
complete code excerpts. Just google for these articles. However,
he does not necessarily always get things right (e.g. I remember a
comparison where he used 'std::istream_iterator
'std::istreambuf_iterator<char>' which has rather different
semantics and is a lot slower).
--
<mailto:dietmar_kuehl (AT) yahoo (DOT) com> <http://www.dietmar-kuehl.de/>
<http://www.contendix.com> - Software Development & Consulting
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Maxim Yegorushkin
Guest
|
| Posted: Sat Mar 19, 2005 12:08 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
SIgnOff <SignOff (AT) gala (DOT) net> wrote:
| Quote: | What the fastest way to copy strings from file to a vector in STL?
|
It depends on how your strings are delimited in the file. For whitespace
delimited strings it can be as simple as:
vector<string> v( // note the addintional parenthesis
(istream_iterator<string>(cin))
, (istream_iterator<string>())
);
If you want to treat lines as strings it can be:
vector<string> v;
string t;
while(getline(cin, t))
v.push_back(t);
If you need absolute raw performance, it surely will be much faster to map
the file in memory and populate the vector with ranges [char const*, char
const*) (boost::iterator_range<char const*> will suffice) rather than
strings. This way you bypass the processing done by file and/or stream and
a great deal of memory allocations done by string objects.
--
Maxim Yegorushkin
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Maciej Sobczak
Guest
|
| Posted: Sat Mar 19, 2005 12:09 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
Hi,
SIgnOff wrote:
| Quote: | What the fastest way to copy strings from file to a vector in STL?
|
What do you mean by "fastest"?
Fastest to write or fastest to execute?
This is almost idiomatic:
ifstream file("file.txt");
vector<string> v;
string line;
while (getline(file, line))
v.push_back(line);
if (file.eof())
{
// OK, the whole file was read
}
BTW: check also:
http://www.msobczak.com/prog/fastreams/
--
Maciej Sobczak : http://www.msobczak.com/
Programming : http://www.msobczak.com/prog/
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Bronek Kozicki
Guest
|
| Posted: Sat Mar 19, 2005 12:12 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
SIgnOff <SignOff (AT) gala (DOT) net> wrote:
| Quote: | What the fastest way to copy strings from file to a vector in STL?
|
The fastest code is the one that is never executed - so do not copy it
at all. What's your goal? Maybe you can treat file itself (or rather its
cache memory) as a container?
B.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Adrian
Guest
|
| Posted: Sat Mar 19, 2005 12:16 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
1. memory mapped file
2. resize all needed
3. memcpy
"SIgnOff" <SignOff (AT) gala (DOT) net> wrote
| Quote: | What the fastest way to copy strings from file to a vector in STL?
|
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
kanze@gabi-soft.fr
Guest
|
| Posted: Sat Mar 19, 2005 12:18 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
SIgnOff wrote:
| Quote: | What the fastest way to copy strings from file to a vector in STL?
|
In what circomstances? With what implementation, on what
machine? The answer will obviously depend on a number of
factors, which we don't know about.
Also, of course, the differences between the various STL
methodes will typically be minor, not worth worrying about.
Just use whichever seems most natural under the circomstances.
And count on the fact that if there are performance problems,
you're likely to have to go outside of the STL to solve them,
probably with implementation specific solutions like mmap.
--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Alex Vinokur
Guest
|
| Posted: Sat Mar 19, 2005 10:45 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
"SIgnOff" <SignOff (AT) gala (DOT) net> wrote
| Quote: | What the fastest way to copy strings from file to a vector in STL?
[snip] |
There are 3 ways to copy file to a vector:
1) one file's line -> one string
2) one file's word -> one string
3) one file's char -> one char.
Sample.
--- File foo ---
ABC DE
XYZ RT MN
----------------
1) Way-1
vector<string> v1;
v1[0] = "ABC DE";
v1[1] = "XYZ RT MN".
2) Way-2
vector<vector v2;
v2[0][0] = "ABC";
v2[0][1] = "DE";
v2[1][0] = "XYZ";
v2[1][1] = "RT";
v2[1][2] = "MN".
3) Way-3
vector<char> v3;
v3[0] = "A";
v3[1] = "B";
v3[2] = "C";
v3[3] = " ";
v3[4] = "D";
....
v3[14] = "N".
What way do you mean?
--
Alex Vinokur
email: alex DOT vinokur AT gmail DOT com
http://mathforum.org/library/view/10978.html
http://sourceforge.net/users/alexvn
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Steven E. Harris
Guest
|
| Posted: Sun Mar 20, 2005 1:52 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
"Maxim Yegorushkin" <e-maxim (AT) yandex (DOT) ru> writes:
| Quote: | If you want to treat lines as strings it can be:
vector<string> v;
string t;
while(getline(cin, t))
v.push_back(t);
|
It's worth noting that this approach copies each line read upon
insertion into the vector, with the advantage that the memory
allocated in the string "t" gets reused throughout the loop.
Alternately, one could push an empty string into the vector (less
copying), and call getline() on v.back(), thus reading directly into
the most recently inserted string. This way, though, memory gets
reallocated progressively for each line read, which may turn out to be
slower than copying each string twice.
--
Steven E. Harris
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
kanze@gabi-soft.fr
Guest
|
| Posted: Mon Mar 21, 2005 9:48 pm Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
Steven E. Harris wrote:
| Quote: | "Maxim Yegorushkin" <e-maxim (AT) yandex (DOT) ru> writes:
If you want to treat lines as strings it can be:
vector<string> v;
string t;
while(getline(cin, t))
v.push_back(t);
It's worth noting that this approach copies each line read
upon insertion into the vector, with the advantage that the
memory allocated in the string "t" gets reused throughout the
loop.
Alternately, one could push an empty string into the vector
(less copying), and call getline() on v.back(), thus reading
directly into the most recently inserted string. This way,
though, memory gets reallocated progressively for each line
read, which may turn out to be slower than copying each string
twice.
|
It all depends on the implementation. If the profiler says you
have a performance problem with this, do something about it (and
be aware that the changes you make on one system to speed things
up may slow them down on another system.) If not, leave
whatever seems most natural and readable.
And of course, regardless of the implementation and the
technique used, it will have to allocate new memory for each
string.
--
James Kanze GABI Software
Conseils en informatique orientée objet/
Beratung in objektorientierter Datenverarbeitung
9 place Sémard, 78210 St.-Cyr-l'École, France, +33 (0)1 30 23 00 34
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Adrian
Guest
|
| Posted: Mon Mar 21, 2005 10:06 pm Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
| Quote: | The fastest code is the one that is never executed - so do not copy it
at all. What's your goal? Maybe you can treat file itself (or rather its
cache memory) as a container?
|
off course file contains 0 terminated strings, I'm sure :)
memory copying is still faster and faster.
there is no real optimization on copying if data doesn't exceed gigabytes.
only optimization which is considerable is allocation optimization and off
course memory mapped file.
"Bronek Kozicki" <brok (AT) rubikon (DOT) pl> wrote
| Quote: | SIgnOff <SignOff (AT) gala (DOT) net> wrote:
What the fastest way to copy strings from file to a vector in STL?
The fastest code is the one that is never executed - so do not copy it
at all. What's your goal? Maybe you can treat file itself (or rather its
cache memory) as a container?
B.
|
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Bronek Kozicki
Guest
|
| Posted: Tue Mar 22, 2005 11:13 pm Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
Adrian <makaron (AT) makaron (DOT) pl> wrote:
| Quote: | gigabytes. only optimization which is considerable is allocation
optimization and off course memory mapped file.
|
memory mapped files is what I would use here, assuming platform offers
such facility (Windows and flavours of Unix do). Hmmm, maybe this would
be worthy to take a look
http://lists.boost.org/MailArchives/boost/msg77457.php
B.
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
Seungbeom Kim
Guest
|
| Posted: Tue Mar 22, 2005 11:43 pm Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
Maxim Yegorushkin wrote:
| Quote: | If you want to treat lines as strings it can be:
vector<string> v;
string t;
while(getline(cin, t))
v.push_back(t);
|
How about this?
struct string_line
{
std::string str;
operator std::string() const { return str; }
};
std::istream& operator>>(std::istream& is, string_line& sl)
{ std::getline(is, sl.str); return is; }
std::vector<std::string> read_lines(std::istream& is)
{
return std::vector<std::string>(
(std::istream_iterator<string_line>(is)),
std::istream_iterator<string_line>()
);
}
--
Seungbeom Kim
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
SIgnOff
Guest
|
| Posted: Fri Mar 25, 2005 12:56 am Post subject: Re: What the fastest way to copy strings from file to a vect |
|
|
I mean you second choice :-)
2) Way-2
vector<vector v2;
v2[0][0] = "ABC";
v2[0][1] = "DE";
v2[1][0] = "XYZ";
v2[1][1] = "RT";
v2[1][2] = "MN".
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
|
|
| Back to top |
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
|
FAQ
Search
Memberlist
Usergroups
Register
Profile
Log in to check your private messages
Log in
