>Ok I understand the mathematics of how the apparent space loss occurs.
>However I've understood from you that the free sector space can be used
>by other packages, right (even with large packages)?
Yes. Except that it doesn't within the same transaction, and this is
the reason why a 1.3 MB package took so much room. If the system
creates two objects of 1 byte each within the same transaction, I
won't put them in the same sector. I'm going to change this (see
below).
Our discussion made me think that I should change the arithmetics for
the next release to either a function of the real used size and the
occupied sectors, or just the real used size.
The original idea of only mentionning free sectors was that the card
full error which happens when there is no free sector left (for
transaction purposes) also happened when the used size is very close
to the total size. But if I reserve a pool of sectors for transaction
purposes only (this pool will only be used when the card will be
full), I think that it's no longer necessary.
>OK. Understood. Deleting the package creates a copy before deleting it.
>Even though the package is 1.3MB it uses up 2.2MB of free sectors. So
>you'd need 4.4MB of total space to delete it. But the store is only 4MB.
Exactly.
However, I think that I should update the size of the sectors where
objects have been deleted. This means that the sector can be used for
non data purposes (i.e. be a copy sector or a transaction list sector)
Abortion in that case isn't a problem because the list of sectors in
transactions are processed in reverse order and transaction list
sectors are freed when commiting/aborting.
In the first case (copy sector) we could have the following
operations to delete object A (which is on two sectors and it's the
only object on both)
* delete subobject A1
** copy sector 2 to 4 which is free
** add (2,4) to the transaction list (which is on sector 1)
** mark sector 4 as non data
** empty sector 2
** mark sector 2 as empty
* delete subobject A2
** copy sector 3 to 2 which is free
** add (3,2) to the transaction list (still on sector 1)
** mark sector 2 as non data
** empty sector 3
** mark sector 3 as empty
Let's say that something wrong happens now. So the system asks us to
abort the transaction. We'll have:
* copy sector 2 to 3
* mark sector 2 as empty
* update size of sector 3
* remove (3,2) from the transaction list
* copy sector 4 to 2
* mark sector 4 as empty
* update size of sector 2
* remove (2,4) from the transaction list
Same logic happens if the transaction list exceeds sector 1 and I
allocate a new sector for it which could be sector 2:
* delete subobject A1
** copy sector 2 to 4 which is free
** and add (2,4) to the transaction list (which is on sector 1)
** mark sector 4 as non data
** empty sector 2
** mark sector 2 as empty
* delete subobject A2
** copy sector 3 to 2 which is free, and add (3,2) to the transaction list
** mark sector 2 as non data
** empty sector 3
** mark sector 3 as empty
Doing this immediate size updating (i.e. marking the size of the
sectors as soon as I'm finished with them, vs doing it at the end of
the transaction) will improve sector usage. It will also fasten the
process as I won't have to compute the size of the sectors at the end
of the transaction. But it has a big drawback (and this is why I
haven't adopted this strategy). It's with separated abortions.
Let's suppose the following operations within a single transaction: I
delete an object and then I create a new one. I can't give this new
one the same ID or if the system wants a separate abortion on the
first object, I'll have a conflict. Let's say that I give it another
ID (which means maintaining a list of deleted sectors - but should
this list be in memory with the risk that it will be too big to fit
or should it be on the store with the risk that it will take room and
slow down things and that I should write where it is to dispose it if
the plug is removed?), and the second object takes the whole sector,
if the system wants a separated abortion on the first object, I'll
have to resize the other one. In a nutshell, all this will be fun to
code :)
This is the kind of problems I enjoyed (er) coding and I was quite
pleased that the transaction engine works more or less without a
trouble. Your experience underlines that it's not ready for shipping
yet. BTW, fixing bugs in the transaction engine took me weeks,
because I had to make big logs, create situations to make the bugs
occur, analyze the sectors by hand and so on. Debug code for that
(which I never included in the releases) represents a third of the
total code.
>Indeed. The message about restoring the store to a clean state never
>occurs.
Heh. Actually, the Newton was locked because it was aborting the
transaction and this failed.
>OUCH! Sorry I've been such a showstopper. Thanks for all the effort you
>are putting in to fix this.
Actually, it's more a matter to improve the software than fixing just
what may seem to be a minor bug for extreme situations. I have to fix
it.
>Intercept deletion on ATA stores. If free space remaining is less than
>that required by the journal then stop the deletion from occurring.
This isn't possible technically. I can't intercept what the system is
doing, and the first deletion when deleted the package occurred after
more than 80% of the job was done. Before, the system wrote stuff,
created new objects and other things for his own fun. All that within
a single transaction, of course.
>Anything more can be ignored for the ATA 1.0 release. [For a post-ATA
>release you could investigate providing the option of disabling the
>journal for the period of the transaction].
This isn't possible neither. It will lead to data corruption for sure
with no way to recover it. The reason is again that I don't know what
the system stores in all these objects. All I do is handling objects.
>I understand why the package cannot be deleted. The workaround is:
>
>* use large flash memory stores. For example use a full 32MB for a 32MB
>card. 32MB is now the low end of the Compact Flash market.
Heh. Personally, I have a 16 MB card. And Victor did some tests for
me with his 4 MB card. Even if CF cards are cheap compared to linear
cards, this isn't a solution.
BTW, I think that linear stores won't allow you to store a package if
there isn't enough room left to have a copy of it. Also, they
probably don't show the actual total size (I think that they need at
least one 64 KB block for erasal). I chose to not reserve a fixed
pool of sectors for transaction purposes because this way, I never
write always the same sectors (except for the map actually). And
sectors can technically be anywhere except the first sector which is
rarely written (the map doesn't have to be the second sector).
>Let the user to set a per-partition reserved sector size. Make it a
>default of 100 sectors (allow a 4-5 digit selection box). Don't allow
>packages to be stored on the reserved sectors but do allow the reserved
>sectors to be used for deletion operations (and if this is no possible
>then add a tick box to allow the reserve sectors to be disabled in the
>preferences).
>
>Cautious users may want to reserve a large number of sectors. Others may
>want to pack as much on the card as possible.
This is indeed a solution, i.e. put this in the preferences. However,
how many advanced users will want to tune this setting?
>So long as there are enough small packages stored on the card it should
>be possible to delete enough of them in order to be able to delete the
>larger packages.
Exactly. This is true for other data than packages, BTW. I think that
the system does a single transaction for each entry.
>I think (b) is fine.
[(b) is doing transaction only on metadata]
>I don't understand why (b) is not acceptable to you.
Simple. I don't want to lose my data :)
>Journalling filesystems typically implement (b), like ReiserFS that
>I am using now.
Yes. But here, we do not have a file system. It's a database. I asked
to Landon Dyer and he told me that linear stores do full journalizing.
>Otherwise there is a huge performance hit if all data is being
>duplicated.
Yes. It's the price of full journalizing. But on the other hand, data
corruptions are rare. For a personal computer, especially considering
that backup solutions are poor, it's a good idea.
Paul
-- Home page: http://www.kallisys.com/ Newton-powered WebServer: http://newt.dyndns.org:8080/-- This is the Newtontalk mailinglist - http://www.newtontalk.net To unsubscribe or manage: visit the above link or mailto:newtontalk-request_at_newtontalk.net?Subject=unsubscribe
This archive was generated by hypermail 2.1.2 : Thu Nov 01 2001 - 10:02:23 EST