Professional Web Applications Themes

How long does DD take? 180GB? - Linux Setup, Configuration & Administration

I had a 360GB RAID (2 X WD 1800JB) and one of the harddrives started acting funny, and thus my RAID went offline. So I got 2 replacement drives, and I am in the process of using DD to copy the entire harddrive to the replacement harddive, (X 2) and hopefully, I will just put the 2 replacement harddrives back into my computer, and it will work (cross fingers, ~330GB of data is not fun to loose) Both harddrives check out fine in WD's driver utilities. For now, which is good. Anyways, so I am using DD to copy the ...

  1. #1

    Default How long does DD take? 180GB?

    I had a 360GB RAID (2 X WD 1800JB) and one of the harddrives started
    acting funny, and thus my RAID went offline. So I got 2 replacement
    drives, and I am in the process of using DD to copy the entire
    harddrive to the replacement harddive, (X 2) and hopefully, I will
    just put the 2 replacement harddrives back into my computer, and it
    will work (cross fingers, ~330GB of data is not fun to loose) Both
    harddrives check out fine in WD's driver utilities. For now, which is
    good. Anyways, so I am using DD to copy the entire partition. I
    booted to linux from CDRom, and use the DD command to copy one
    harddrive to the other. Here I am sitting, it's 16 hours + into the
    operation, and it's not done. I have the source HDD on secondard
    slave, and the destination HDD on primary secondary. 180GB / 16 hours
    it about 3 megs/second, minimum. It's still not done. Is this
    normal? If I would have installed Linux on a HDD would this go
    faster? Is DMA not installed? The harddrive light is still lit up, it
    sounds like it's copying, I can feel each harddrive doing it's thing,
    it totally looks to be copying. It's just taking a smegload of a
    lotta time. I am wondering if I am doing this right, or how I can
    speed this up for next time. Again, it appears that it's copying,
    nothing is locked up, etc. I'm not doing anything to take up
    resources, etc. Help!
    Frank Guest

  2. #2

    Default Re: How long does DD take? 180GB?

    Frank Burns wrote:
     

    Did you run this on both drives:

    # hdparm -Tt /dev/hd(a-d) (tests drive r/w speeds)
     

    Do the math. 3 megabytes per second, 360 GB total, assuming the drive is
    full or you are copying an image (you are, since you are using dd). That
    is:

    360e9/3e6 = 120,000 seconds = 33 1/3 hours. This doesn't take read time into
    account, it could be double this value.
     

    If you had only copied the files, not the disk image, it would be faster
    unless the drive is completely full.
     

    Why are you asking us? Run this on the HDD:

    $ hdparm /dev/hd(a-d) (lists setup parameters)

    And a HDD without DMA enabled can easily be reduced to 3 MB/s.

    --
    Paul Lutus
    http://www.arachnoid.com

    Paul Guest

  3. #3

    Default Re: How long does DD take? 180GB?

    Hello,
    I think your problem is related to dma (man hdparm). To speed it up even
    further, you should take a block size of e.g. 32Mb:
    dd if=/dev/hdXn of=/dev/hdYn bs=32M

    Wolfgang Guest

  4. #4

    Default Re: How long does DD take? 180GB?

    It did finally finish. it said:

    dd: reading '/dev/hdb': Input/output error
    351,651,888+0 records in
    351,651,888+0 records out
    180,045,766,656 byes transferd in 98850.692037 seconds (1,821,391
    byes/sec)

    I don't exactly understand that, or what the error was, I hope it was
    relating to the fact that all partition programs see the 1/2 raid as
    ~360GB in each 180GB HDD.

    So it was about 27 hours for 180GB. Now I gotta do it again. I will
    try your suggestions and see how long it takes.

    (P4 2.6Ghz HT, 1GB DDR400 Dual Channel RAM, WD 1800JB 8 meggacache,
    MSI865PE-Nero2-FSIR)








    Paul Lutus <zzz> wrote in message news:<supernews.com>... 
    >
    > Did you run this on both drives:
    >
    > # hdparm -Tt /dev/hd(a-d) (tests drive r/w speeds)

    >
    > Do the math. 3 megabytes per second, 360 GB total, assuming the drive is
    > full or you are copying an image (you are, since you are using dd). That
    > is:
    >
    > 360e9/3e6 = 120,000 seconds = 33 1/3 hours. This doesn't take read time into
    > account, it could be double this value.

    >
    > If you had only copied the files, not the disk image, it would be faster
    > unless the drive is completely full.

    >
    > Why are you asking us? Run this on the HDD:
    >
    > $ hdparm /dev/hd(a-d) (lists setup parameters)
    >
    > And a HDD without DMA enabled can easily be reduced to 3 MB/s.[/ref]
    Frank Guest

  5. #5

    Default Re: How long does DD take? 180GB?

    In article <google.com>,
    com (Frank Burns) writes: 

    This is verry slow. I think there is something wrong.
    I get with my old cheap duron750 this:

    gar:~ # hdparm -t /dev/hda1

    /dev/hda1:
    Timing buffered disk reads: 32 MB in 0.97 seconds =32.99 MB/sec

    --the settings:

    gar:~ # hdparm /dev/hda

    /dev/hda:
    multcount = 16 (on)
    I/O support = 1 (32-bit)
    unmaskirq = 1 (on)
    using_dma = 1 (on)
    keepsettings = 0 (off)
    nowerr = 0 (off)
    readonly = 0 (off)
    readahead = 8 (on)
    geometry = 9733/255/63, sectors = 156368016, start = 0


    --
    MFG Gernot
    Gernot Guest

  6. #6

    Default Re: How long does DD take? 180GB?

    Frank Burns wrote:
     

    *WHY* are you copying them this way? Why not partition the new RAID
    array, create a file system, and use something like "tar" or "rsync" to
    duplicate the data?

    Doing this by duplicating every byte is like replacing every space on a
    web page with "&nbsp;": it's an amazingly long waste of your time and
    gains you nothing.

    Nico Guest

  7. #7

    Default Re: How long does DD take? 180GB?

    Nico Kadel-Garcia wrote:

    < snip >
     

    Yes, I pointed this out to him yesterday. Something funny going on here.
    Copying partition images is fraught with other risks, not just time wasted.

    --
    Paul Lutus
    http://www.arachnoid.com

    Paul Guest

  8. #8

    Default Re: How long does DD take? 180GB?

    Nico Kadel-Garcia <net> writes:
     
     [/ref]
    {snip}
     
     

    I would agree. On the other hand, copying windows partitions this way is
    something that is definitely _not_ a waste of time. It can be surprisingly
    useful..

    180GB, though, wow, that's unimaginably horrible.
    cheers,
    denice
    --
    denice.deatrich epfl.ch, DSC / LTHC-LTHI, E.P.F.L. PH: +41 (21) 693 76 67
    <*> This moment's fortune cookie:
    Our houseplants have a good sense of humous.
    Denice Guest

  9. #9

    Default Re: How long does DD take? 180GB?

    > *WHY* are you copying them this way? Why not partition the new RAID 

    I can't copy anything from the old RAID-0 because it's not working.

    I can't have 2 raid 0's in 1 system with only 1 raid controller, and
    one raid wouldn't even work, so I couldn't even get any data off it.
    This is the only way I can do it. copy each drive, and try to recreate
    the raid.
    Frank Guest

  10. #10

    Default Re: How long does DD take? 180GB?

    Denice DEATRICH wrote:
     
     
    >
    >
    > I would agree. On the other hand, copying windows partitions this way is
    > something that is definitely _not_ a waste of time. It can be surprisingly
    > useful..
    >
    > 180GB, though, wow, that's unimaginably horrible.
    > cheers,
    > denice[/ref]

    No, no, horribly unimaginable is installing machines by burning
    different "dd" based disk images if the machine is 8 Gig or 10 Gig,
    including the swap space, and trying to put them both on the same CD
    requiring the user to hit a keyboard entry to select what size the disk
    was at boot time.

    *THAT* was unimaginably horrible....

    Nico Guest

  11. #11

    Default Re: How long does DD take? 180GB?

    Frank Burns wrote:
     
    >
    >
    > I can't copy anything from the old RAID-0 because it's not working.
    >
    > I can't have 2 raid 0's in 1 system with only 1 raid controller, and
    > one raid wouldn't even work, so I couldn't even get any data off it.
    > This is the only way I can do it. copy each drive, and try to recreate
    > the raid.[/ref]

    Ohh. *OUCH*, that s. Couldn't you buy or borrow another RAID
    controller to speed this process?

    Nico Guest

  12. #12

    Default Re: How long does DD take? 180GB?


    Frank Burns wrote: 

    I realise that my remark won't help you in your current predicament (for
    which I apologise), but the above reason is why I went for software RAID
    rather than use a RAID controller.

    My requirement was for redundancy rather than speed, so I used RAID1. I
    didn't want the RAID controller to still be a single point of failure,
    which if it happened meant that I had to find another compatible RAID
    controller in order to recover my data. Hence software RAID.

    Had my requirement been for speed in addition to or instead of
    redundancy I would probably still have gone for software RAID because of
    the flexibility that I think it brings: various kinds of failure,
    including failure of the controller or even the motherboard, can be
    recovered from by replacing the failed component with any bog standard
    spare. I realise that there is a price to pay in terms of reduced
    throughput (or increased bus usage) as compared with 'hardware' RAID,
    but for my purposes such considerations would not prevail.

    Back to your situation: could I get you to reconsider whether you don't
    also want redundancy in addition to speed? That is: move to RAID5 rather
    than RAID0, even if you need to do it using a dedicated controller? Do
    you really want to have 360GB of data without any redundancy?

    Once again: apologies if my remarks appear to be counterproductive, that
    is not how they are intended.

    Jan
    Jan Guest

  13. #13

    Default Re: How long does DD take? 180GB?

    > Ohh. *OUCH*, that s. Couldn't you buy or borrow another RAID 

    Okay here is the status & other problems:
    I used this command (after I enabled DMA (3.47MB/sec vs 47.9MB/sec))

    dd if=/dev/hda of=/dev/hdb bs=32M

    it said:

    171705+1 records in
    171705+1 records out
    180,045,766,656 bytes
    35822.125322 seconds (I think it was about 2 hours in reality, it
    can't add right)
    5,025,105 bytes/sec

    (notice the records are different, for identical drives)

    Now I didn't think I could use another RAID controller, (can I???)
    Whenever I try to reboot on the Old RAID it says "Raid Offline" so I
    can get into it, can I? If I get another raid controller, can I
    install my new raid on my motherboard, the old raid on the controller
    and copy the old raid to the new raid?

    This is an entire 1 partition WinXP SP1 Partition NTFS 5.1, (I know, I
    know) on this computer.

    I copied the two drives, and remember I did get an error message when
    I copied the first drive, I put them back in, and they didn't work.
    The RAID bios said the two drives were FREE and not in an array. So
    does DD not copy everything? the raid info is stored on the HDD, not
    the raid bios. Do I need to go to a DATA restorer guy and see if I can
    use a physical HDD copier?

    Help!

    Raid s. I dislike raid 0 for obvious reasons, but it is so
    noticably faster. I dislike raid 1 because if one OS goes south,
    you're ed, twice over. But it's better. I guess RAID 0,1 or 5 is
    best. but I'm not made of money... maybe I should be.
    Frank Guest

  14. #14

    Default Re: How long does DD take? 180GB?

    > Ohh. *OUCH*, that s. Couldn't you buy or borrow another RAID 

    Okay here is the status & other problems:
    I used this command (after I enabled DMA (3.47MB/sec vs 47.9MB/sec))

    dd if=/dev/hda of=/dev/hdb bs=32M

    it said:

    171705+1 records in
    171705+1 records out
    180,045,766,656 bytes
    35822.125322 seconds (I think it was about 2 hours in reality, it
    can't add right)
    5,025,105 bytes/sec

    (notice the records are different, for identical drives)

    Now I didn't think I could use another RAID controller, (can I???)
    Whenever I try to reboot on the Old RAID it says "Raid Offline" so I
    can get into it, can I? If I get another raid controller, can I
    install my new raid on my motherboard, the old raid on the controller
    and copy the old raid to the new raid?

    This is an entire 1 partition WinXP SP1 Partition NTFS 5.1, (I know, I
    know) on this computer.

    I copied the two drives, and remember I did get an error message when
    I copied the first drive, I put them back in, and they didn't work.
    The RAID bios said the two drives were FREE and not in an array. So
    does DD not copy everything? the raid info is stored on the HDD, not
    the raid bios. Do I need to go to a DATA restorer guy and see if I can
    use a physical HDD copier?

    Help!

    Raid s. I dislike raid 0 for obvious reasons, but it is so
    noticably faster. I dislike raid 1 because if one OS goes south,
    you're ed, twice over. But it's better. I guess RAID 0,1 or 5 is
    best. but I'm not made of money... maybe I should be.
    Frank Guest

Similar Threads

  1. Question about a long session timeout (somewhat long)
    By Stupid48 in forum ASP.NET Security
    Replies: 7
    Last Post: March 1st, 10:04 PM
  2. IDS 7.3* - Long long long checkpoint !
    By Laurent in forum Informix
    Replies: 1
    Last Post: October 28th, 09:18 PM

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •  

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139