>Thanks for the full explanation of storing onto ATA cards. I realise it
>could be different storage used but almost certainly not ~ double (I
>didn't know you could use free bytes within clusters).
I'm not totally happy with the design of ATA stores (I've thought
several times under the shower that it would need to be entirely
redesigned with other choices, but my priority is to release
something working), but at least, it uses most of sectors when the
store becomes full.
>Have you considered that the compression or fast compression bit might
>not be honoured? Perhaps when a large compressed file is uploaded to the
>store it is being uncompressed? But its original compressed size is
>being recorded. Instant corruption.
The compression bit is of course honored as it's in fact not the ATA
Support's job, but the system's. ATA Support doesn't know about
packages, soups and NewtonScript objects. It only knows about store
objects which are often equal to NewtonScript objects, but sometimes
not, and this is what the system calls large objects or large binary
objects. LBOs can be compressed and the system actually split them in
several store-objects. VBOs are just like LBOs with little
differences to include a virtual memory behavior.
Additionally, the ATA store engine splits objects in sub-objects to
make them fit in a sector. This is where ATA stores differs
technically from linear stores. Linear stores are divided in 64 KB
blocks (the minimum unit for writing) and allocates objects in these
blocks. Therefore, their objects are at most 64 KB long. There is a
table of objects somewhere, and there is definitely a loss because of
lack of fragmentation (objects can't be splitted in several blocks).
ATA stores are divided in 512 bytes sectors, and there is no table
(well, except for transaction and in the case of bad sectors). There
is a map, though, indicating the state of each sector (but this map
is written the minimum number of times and is technically always in
cache). From the object ID, ATA Support determines in which sector
the object is (basically by left-shifting of 5 bits the ID). Since a
sector cannot hold what the system wants to store in an object (up to
64 KB), objects are technically divided in several objects (there is
a flag indicating whether the object is fragmented or not to know if
I need to read the next segment ID).
Here is a summary of both methods:
Linear:
* maybe faster - provided that the access times are equal - but I'm
not sure of that, as it requires a single lookup in a table for
objects whatever their size is.
* however, it requires garbage collection from time to time.
* requires to write 64 KB of data even if you only change 1 byte.
ATA:
* uses most of the available space (although the structure data may
be larger than with linear stores) without garbage collection.
* for small objects, there is no lookup at all, but for large objects
(> 510 bytes if there are free sectors left, any size if the store is
nearly full), there may be more lookups. Only write the minimum
number of sectors. Resizing an object (which is a frequent operation)
technically means only writing the map sector and the sector of the
last segment and maybe another one - plus transaction operations.
I'm not sure whether linear method would be faster. What you should
know is that the store system has been built with linear cards (or
internal memory) limitations in mind: feel free to read often as it
doesn't cost anything, data blocks shouldn't be bigger than 64 KB,
etc.
Optimization of ATA stores will definitely be a research direction
for ATA Support 1.1.
Paul
-- Home page: http://www.kallisys.com/ Newton-powered WebServer: http://newt.dyndns.org:8080/-- This is the Newtontalk mailinglist - http://www.newtontalk.net To unsubscribe or manage: visit the above link or mailto:newtontalk-request_at_newtontalk.net?Subject=unsubscribe
This archive was generated by hypermail 2.1.2 : Thu Nov 01 2001 - 10:02:19 EST