Re: [NTLK] Experiencing Serious ATA 1.0b8-D Data Corruption ...

From: Adam Warner (lists_at_consulting.net.nz)
Date: Sun Oct 21 2001 - 09:42:11 EDT


On Mon, 2001-10-22 at 01:13, Paul Guyot wrote:
>
> >Ok I understand the mathematics of how the apparent space loss occurs.
> >However I've understood from you that the free sector space can be used
> >by other packages, right (even with large packages)?
>
> Yes. Except that it doesn't within the same transaction, and this is
> the reason why a 1.3 MB package took so much room. If the system
> creates two objects of 1 byte each within the same transaction, I
> won't put them in the same sector. I'm going to change this (see
> below).

Then I'm glad you will be changing this. I'm also glad that our
discussions are helping you to think (out loud).

I suspect any change may require people to reformat their stores. Would
you really want to support reading (and writing to the the free space)
of the current storage format?

<scary technical discussion snipped>

> This is the kind of problems I enjoyed (er) coding and I was quite
> pleased that the transaction engine works more or less without a
> trouble. Your experience underlines that it's not ready for shipping
> yet. BTW, fixing bugs in the transaction engine took me weeks,
> because I had to make big logs, create situations to make the bugs
> occur, analyze the sectors by hand and so on. Debug code for that
> (which I never included in the releases) represents a third of the
> total code.

It really sucks that you're going to have to rewrite (some of) it.
 
> >OUCH! Sorry I've been such a showstopper. Thanks for all the effort you
> >are putting in to fix this.
>
> Actually, it's more a matter to improve the software than fixing just
> what may seem to be a minor bug for extreme situations. I have to fix
> it.

OK. Your ethics and high standards mean you have no other choice :-)

> >Intercept deletion on ATA stores. If free space remaining is less than
> >that required by the journal then stop the deletion from occurring.
>
> This isn't possible technically.

OK. That's why I also suggested the user-definable reserved sectors
"workaround".

> >I understand why the package cannot be deleted. The workaround is:
> >
> >* use large flash memory stores. For example use a full 32MB for a 32MB
> >card. 32MB is now the low end of the Compact Flash market.
>
> Heh. Personally, I have a 16 MB card. And Victor did some tests for
> me with his 4 MB card. Even if CF cards are cheap compared to linear
> cards, this isn't a solution.

[Which means 1.3MB packages that use up 2.2MB of space isn't a solution
either].

> >Let the user to set a per-partition reserved sector size. Make it a
> >default of 100 sectors (allow a 4-5 digit selection box). Don't allow
> >packages to be stored on the reserved sectors but do allow the reserved
> >sectors to be used for deletion operations (and if this is no possible
> >then add a tick box to allow the reserve sectors to be disabled in the
> >preferences).
> >
> >Cautious users may want to reserve a large number of sectors. Others may
> >want to pack as much on the card as possible.
>
> This is indeed a solution, i.e. put this in the preferences. However,
> how many advanced users will want to tune this setting?

Not only in the preferences. Non-default values would have to be stored
on partitions themselves to allow users to vary the reserved sectors
depending on the size of the partition and the number and size of
packages stored. [Some paritions could be "read only" and users may want
to squeeze as much on those partitions without the possibility to delete
packages].

I agree it is somewhat messy. But you're already performed miracles. If
user-defined reserved sectors are indeed necessary (with a conservative
dynamic default that could be equal to the largest package stored on the
card) then so be it. The Newton platform can't be everything we want it
to be because Apple abandoned it.

How many advanced users want to have to worry about -10061 errors, or
running out of heap, or having to reset their Newton because they opened
up the Extras draw and clicked a small utility while running the Waba
version of Pac-Man? [Waba is so awesome. I discovered it yesterday. It's
given me the incentive to upgrade my 1MB of RAM].

There are limitations we're going to have to learn to live with. However
I would like to see more efficient storage in these situations (which
goes some way to resolving the amount of space required to delete
packages).

> >So long as there are enough small packages stored on the card it should
> >be possible to delete enough of them in order to be able to delete the
> >larger packages.
>
> Exactly. This is true for other data than packages, BTW. I think that
> the system does a single transaction for each entry.
>
> >I think (b) is fine.
>
> [(b) is doing transaction only on metadata]
>
> >I don't understand why (b) is not acceptable to you.
>
> Simple. I don't want to lose my data :)
>
> >Journalling filesystems typically implement (b), like ReiserFS that
> >I am using now.
>
> Yes. But here, we do not have a file system. It's a database. I asked
> to Landon Dyer and he told me that linear stores do full journalizing.

OK. Thanks for all the info.

Regards,
Adam

--
This is the Newtontalk mailinglist - http://www.newtontalk.net
To unsubscribe or manage: visit the above link or
	mailto:newtontalk-request_at_newtontalk.net?Subject=unsubscribe



This archive was generated by hypermail 2.1.2 : Thu Nov 01 2001 - 10:02:23 EST