1.1 --- a/docs/wiki/Components Sat Jun 11 18:29:28 2022 +0200
1.2 +++ b/docs/wiki/Components Sun Jun 12 00:00:12 2022 +0200
1.3 @@ -251,6 +251,7 @@
1.4
1.5 {{{
1.6 mmap(in offset_t position, in offset_t length,
1.7 + in offset_t start_visible, in offset_t end_visible,
1.8 out offset_t start_pos, out offset_t end_pos,
1.9 out offset_t size)
1.10 }}}
2.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
2.2 +++ b/docs/wiki/Directories Sun Jun 12 00:00:12 2022 +0200
2.3 @@ -0,0 +1,273 @@
2.4 += Directories =
2.5 +
2.6 +Filesystem directories contain collections of files, other directories, along
2.7 +with other filesystem objects. Directories can be listed, permitting
2.8 +filesystem clients to peruse the contents of each directory. However, the act
2.9 +of listing a directory may impose certain constraints on the operations being
2.10 +conducted on the directory during the listing process.
2.11 +
2.12 +Operations that affect directory contents include the following:
2.13 +
2.14 + * Creating a new object
2.15 + * Removing an object
2.16 + * Moving an object out of the directory
2.17 + * Moving an object into the directory
2.18 +
2.19 +It is conceivable that for some filesystems, these operations will not be able
2.20 +to rely on directly modifying the directory since a listing operation may be
2.21 +in progress. Consequently, a number of rules would be imposed for such
2.22 +filesystems. It appears that Ext2-based filesystems accessed via libext2fs do
2.23 +not require such rules to be applied outside the library due to the design of
2.24 +the filesystem structures and the behaviour of the library itself.
2.25 +
2.26 +== Providers ==
2.27 +
2.28 +Providers of filesystem objects are created when at least one filesystem
2.29 +client is accessing such an object, with the object regarded as being
2.30 +'''open'''. By definition, a provider will exist until all clients have closed
2.31 +or discarded their means of accessing the object.
2.32 +
2.33 +An object is regarded as '''accessible''' if it can still be opened by
2.34 +filesystem clients. Where an object has been removed, it will no longer be
2.35 +accessible since clients should not be able to see the object in the
2.36 +filesystem any more. (The actual filesystem residing in storage may not have
2.37 +been updated for such a removal, this being the eventual outcome of a removal
2.38 +operation.)
2.39 +
2.40 +Providers can be regarded as '''accessible''' if they maintain access to
2.41 +objects that are open and accessible. Clients opening objects will only gain
2.42 +access to accessible providers. Providers can become inaccessible if an object
2.43 +is removed even if the object (and its provider) is still open.
2.44 +
2.45 +Initially, providers will correspond to objects resident in the stored
2.46 +filesystem. Thus, looking up an object using its path will involve the
2.47 +filesystem, yielding a file identifier that can be used to map to a provider.
2.48 +
2.49 +{{{
2.50 +def get_provider_from_filesystem(path):
2.51 + fileid = opening.get_fileid(path)
2.52 +
2.53 + if not fileid:
2.54 + return error
2.55 +
2.56 + return find_provider(fileid)
2.57 +}}}
2.58 +
2.59 +However, providers may not always immediately correspond to objects resident
2.60 +in the stored filesystem. Where an object is created but cannot be immediately
2.61 +registered in the contents of a directory, it must be registered separately
2.62 +and attempts to open this object must consult this separate mapping of
2.63 +filenames to providers.
2.64 +
2.65 +{{{
2.66 +def get_provider(path):
2.67 + provider = find_created_provider(path)
2.68 +
2.69 + if provider:
2.70 + return provider
2.71 +
2.72 + return get_provider_from_filesystem(path)
2.73 +}}}
2.74 +
2.75 +Providers that are inaccessible need to be retained until they are closed.
2.76 +However, since they are no longer accessible, they should not be available to
2.77 +the provider lookup operations.
2.78 +
2.79 +When providers are closed, they are removed from any mapping in which they
2.80 +could be found. Inaccessible providers that have been retained outside the
2.81 +identifier or filename mappings will represent objects that should be removed
2.82 +from their parent directory. Accessible providers retained by the mappings
2.83 +will represent objects that should be created in their parent directory.
2.84 +
2.85 +Whether object removal or creation will occur depends on whether the parent
2.86 +directory is able to safely perform these operations concurrently with other
2.87 +operations (such as servicing a listing operation) or whether such operations
2.88 +will need to be deferred until they can be safely performed.
2.89 +
2.90 +== Object Removal ==
2.91 +
2.92 +Filesystem object removal involves obtaining any provider of the object. Where
2.93 +a provider can be obtained, it will be made inaccessible and removal will be
2.94 +requested. The actual object will not be removed at least until it is closed
2.95 +because it may still receive and provide data. (Unlinking open files is a
2.96 +feature of Unix systems.)
2.97 +
2.98 +Where a provider cannot be obtained, an attempt will be made to obtain the
2.99 +parent directory provider, and if this succeeds, it indicates that the
2.100 +directory is being accessed and must therefore be notified of the intention to
2.101 +eventually remove the object concerned.
2.102 +
2.103 +Without any provider of the object, and where no directory provider can be
2.104 +obtained, the object can be immediately removed from the filesystem.
2.105 +
2.106 +{{{
2.107 +def remove_object(path):
2.108 + provider = get_provider(path)
2.109 +
2.110 + if provider:
2.111 + return make_provider_inaccessible(provider)
2.112 +
2.113 + dirname, basename = split(path)
2.114 + directory_provider = get_provider(dirname)
2.115 +
2.116 + if directory_provider:
2.117 + return directory_provider.remove_pending(basename)
2.118 +
2.119 + return filesystem.remove_object(path)
2.120 +}}}
2.121 +
2.122 +It should be noted that with no accessible provider, any attempt to create a
2.123 +file with the same name as a removed file should cause a new "version" of the
2.124 +file to be created.
2.125 +
2.126 +== Object Creation ==
2.127 +
2.128 +Filesystem object creation occurs when no existing provider can be found for a
2.129 +named object and where creation is requested. Any new object will be
2.130 +accessible and remain so until or unless it is removed.
2.131 +
2.132 +Whether an object is created in the filesystem storage depends on whether the
2.133 +parent directory is being used.
2.134 +
2.135 +If a parent directory is not being used, the object can be registered in its
2.136 +proper location in the filesystem itself, yielding a file identifier.
2.137 +
2.138 +If a parent directory is being used, the object cannot be immediately
2.139 +registered in its proper location, but since data may still need to be written
2.140 +to storage, a temporary location is employed, yielding a file identifier.
2.141 +
2.142 +For an object created in a temporary location, a temporary mapping from the
2.143 +path of the object to the provider is established, with the parent directory
2.144 +being notified of the pending object creation.
2.145 +
2.146 +{{{
2.147 +def create_object(path):
2.148 + provider = get_provider(path)
2.149 +
2.150 + if provider:
2.151 + return error
2.152 +
2.153 + dirname, basename = split(path)
2.154 + directory_provider = get_provider(dirname)
2.155 +
2.156 + if directory_provider:
2.157 + provider = make_provider(get_temporary_location(path))
2.158 + directory_provider.create_pending(provider)
2.159 + return register_created_provider(path, provider)
2.160 +
2.161 + provider = make_provider(path)
2.162 + return register_provider(provider)
2.163 +}}}
2.164 +
2.165 +== Created Providers ==
2.166 +
2.167 +Created providers are retained by the registry until their files can be
2.168 +committed to the filesystem in the desired location.
2.169 +
2.170 +The `register_created_provider` operation establishes a mapping from a path to
2.171 +a provider that can be queried using the `find_created_provider` operation.
2.172 +
2.173 +Created providers are deregistered when they are closed. This will occur when
2.174 +the parent directory of the file to be committed is closed. At that time, the
2.175 +created file will be moved from its temporary location to the desired
2.176 +location.
2.177 +
2.178 +== Inaccessible Providers ==
2.179 +
2.180 +Inaccessible providers are retained by the registry until they are closed.
2.181 +
2.182 +Where the provided file resides in a non-temporary location, closure will
2.183 +occur when the parent directory of the provided file is closed, this having
2.184 +obstructed the removal of the file. At that time, the provided file will be
2.185 +removed.
2.186 +
2.187 +Where the provided file resides in a temporary location, closure will not be
2.188 +obstructed by any directory and will cause the file to be removed immediately.
2.189 +
2.190 +== Directory Provider Closure ==
2.191 +
2.192 +A critical event that typically initiates housekeeping work for created and
2.193 +removed files is the closure of the parent directory of those files.
2.194 +
2.195 +Directory providers support the `create_pending` and `remove_pending`
2.196 +operations that register the details of affected files. When closure occurs,
2.197 +the provider will contact the registry to initiate the necessary work to
2.198 +relocate created files and to remove files.
2.199 +
2.200 +
2.201 +
2.202 +
2.203 + Where an object provider is notified of pending removal, it will initiate
2.204 + removal of the actual object upon closure, checking for an active parent
2.205 + provider.
2.206 +
2.207 + Where a parent provider is notified of pending removal, it will initiate
2.208 + removal of the actual object upon closure, checking for an active object
2.209 + provider.
2.210 +
2.211 + Object opening logic would need to be adjusted. Consider the case where a
2.212 + file is removed but still open.
2.213 +
2.214 + Any attempt to open a file with the same name must ignore the original and
2.215 + open a new file of that name which would be stored elsewhere until such
2.216 + time as the original is actually removed from its location.
2.217 +
2.218 + This new file might have a different identifier since the original file
2.219 + would still exist and retain the identifier.
2.220 +
2.221 + Any such new, but uncommitted, file must be accessible by other clients
2.222 + attempting to open a file of that name. However, removal operations must
2.223 + also be possible, making this version of the open file also unaccessible
2.224 + to new opening clients.
2.225 +
2.226 + Therefore, any providers used by opening clients must only refer to a
2.227 + version of a file that is pending removal if no other version has been
2.228 + created. Once a new version has been created, any provider pending removal
2.229 + must be relegated to a separate collection.
2.230 +
2.231 + Storage of new versions of files could be confined to special directories
2.232 + that are hidden from clients.
2.233 +
2.234 + The opening logic would be as follows:
2.235 +
2.236 + provider = find_provider(path)
2.237 +
2.238 + if provider:
2.239 + if provider.pending_removal():
2.240 + relegate(provider)
2.241 + provider = create_provider(path, hidden=True)
2.242 + else:
2.243 + provider = create_provider(path, hidden=False)
2.244 +
2.245 + # Have provider.
2.246 +
2.247 + return provider.make_resource()
2.248 +
2.249 + A provider would normally be obtained by consulting the filesystem
2.250 + implementation. However, new versions of files replacing ones pending
2.251 + removal will reside in locations that are different to the intended
2.252 + location of the file. Consequently, the registry needs to maintain its own
2.253 + mapping of paths to providers for such file versions.
2.254 +
2.255 + The opportunity to remove a file definitively arises when the following
2.256 + conditions are satisfied:
2.257 +
2.258 + * The parent directory is not open in some way
2.259 + * The original provider for the file is no longer open
2.260 +
2.261 + For new versions of a removed file that have themselves been marked as
2.262 + pending removal, their closure is sufficient to remove their filesystem
2.263 + resources.
2.264 +
2.265 + However, the registry must maintain a path entry for any active version
2.266 + of a removed file until that version is closed. Thus, the following
2.267 + conditions apply:
2.268 +
2.269 + * The parent directory (of the original file) is not open in some way
2.270 + * The provider of the new version is no longer open
2.271 +
2.272 + It will not be generally possible to open the hidden directory containing
2.273 + new file versions. Therefore, the transfer of the file to its intended
2.274 + location will not risk interfering with any operations on the hidden
2.275 + directory.
2.276 +
3.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
3.2 +++ b/docs/wiki/FilesystemAccess Sun Jun 12 00:00:12 2022 +0200
3.3 @@ -0,0 +1,300 @@
3.4 += Filesystem Access =
3.5 +
3.6 +Within the filesystem server library, a number of different abstractions and
3.7 +mechanisms are employed to provide access to filesystem objects.
3.8 +
3.9 +<<TableOfContents(2,3)>>
3.10 +
3.11 +The abstractions or components used in the library are organised as follows.
3.12 +
3.13 +######## A graph showing the relationships between library components
3.14 +
3.15 +{{{#!graphviz
3.16 +#format svg
3.17 +#transform notugly
3.18 +digraph components {
3.19 + node [fontsize="12.0",fontname="sans-serif",shape=box];
3.20 + edge [fontsize="12.0",fontname="sans-serif"];
3.21 + rankdir=LR;
3.22 +
3.23 + subgraph {
3.24 + rank=same;
3.25 +
3.26 + Resource -> DirectoryResource -> FilePager
3.27 + [dir=none,style=dotted];
3.28 + }
3.29 +
3.30 + subgraph {
3.31 + rank=same;
3.32 +
3.33 + Provider -> DirectoryProvider -> FileProvider
3.34 + [dir=none,style=dotted];
3.35 + }
3.36 +
3.37 + subgraph {
3.38 + rank=same;
3.39 +
3.40 + Accessor -> Ext2FileAccessor
3.41 + [dir=none,style=dotted];
3.42 + }
3.43 +
3.44 + subgraph {
3.45 + rank=same;
3.46 +
3.47 + DirectoryAccessor -> Ext2DirectoryAccessor
3.48 + [dir=none,style=dotted];
3.49 + }
3.50 +
3.51 + subgraph {
3.52 + node [shape=ellipse];
3.53 + rank=same;
3.54 +
3.55 + Directory [label="dir"];
3.56 + File [label="dir/file"];
3.57 + }
3.58 +
3.59 + DirectoryResource -> DirectoryProvider -> Ext2DirectoryAccessor -> Directory;
3.60 + FilePager -> FileProvider -> Ext2FileAccessor -> File;
3.61 +}
3.62 +}}}
3.63 +
3.64 +########
3.65 +
3.66 +This document uses C-style interface syntax since the components operating
3.67 +within the described mechanisms will themselves be described using such
3.68 +syntax.
3.69 +
3.70 +== Accountable ==
3.71 +
3.72 +This interface provides the following operations:
3.73 +
3.74 +{{{
3.75 +void attach()
3.76 +unsigned int detach()
3.77 +}}}
3.78 +
3.79 +Its purpose is to provide reference counting for components, with the `attach`
3.80 +operation incrementing the number of users of a component, and with the
3.81 +`detach` operation decrementing the number of users and returning the
3.82 +resulting number of users. When components no longer have any attached users,
3.83 +they may perform operations to tidy up after themselves and permit their
3.84 +deallocation.
3.85 +
3.86 +== Accessors ==
3.87 +
3.88 +Accessors provide the means of accessing filesystem object data and metadata.
3.89 +Conceptually, files and directories are both exposed by accessors, although
3.90 +the interfaces and mechanisms may differ between these types of objects.
3.91 +
3.92 +=== Directory Accessors ===
3.93 +
3.94 +Currently, directory accessors provide support for traversing directory
3.95 +listings, obtaining the relevant filesystem metadata using the underlying
3.96 +filesystem access library.
3.97 +
3.98 +=== File Accessors ===
3.99 +
3.100 +File content is accessed through an interface with the following operations:
3.101 +
3.102 +{{{
3.103 +void close()
3.104 +void fill(Flexpage *flexpage)
3.105 +void flush(Flexpage *flexpage)
3.106 +offset_t get_size()
3.107 +void set_size(offset_t size)
3.108 +}}}
3.109 +
3.110 +The operations need to be supported with actual filesystem operations. For
3.111 +example, ext2-based filesystems employ a specific abstraction which invokes
3.112 +library functions provided by the `libext2fs` package.
3.113 +
3.114 +== Providers ==
3.115 +
3.116 +Providers encapsulate the essential functionality for accessing filesystem
3.117 +objects. Implementing the `Accountable` interface, they are shared by
3.118 +resources and discarded when no resources are using them.
3.119 +
3.120 +The following operations are supported by providers:
3.121 +
3.122 +{{{
3.123 +ProviderRegistry *registry()
3.124 +long make_resource(offset_t *size, object_flags_t *object_flags, Resource **resource)
3.125 +bool removal_pending()
3.126 +void remove()
3.127 +void remove_pending(bool remove)
3.128 +}}}
3.129 +
3.130 +Providers are associated with filesystem objects in a registry which can be
3.131 +obtained from each provider using the `registry` operation.
3.132 +
3.133 +Providers also support the creation of resources through which each user of a
3.134 +provider exercises its use of that provider. The `make_resource` operation
3.135 +performs the creation and returns size, flags and resource instance details.
3.136 +
3.137 +The removal of providers can be directed using the `remove_pending` and
3.138 +`remove` operations. Where `remove_pending` has been called with a true value
3.139 +as its parameter, the `removal_pending` operation will also return a true
3.140 +value, and upon the provider being discarded, the `remove` operation will be
3.141 +invoked. The `remove` operation performs the appropriate underlying operation
3.142 +to remove the object being provided.
3.143 +
3.144 +Typically, removal of providers is managed by the provider registry when
3.145 +resources are closed and detached from providers.
3.146 +
3.147 +######## A graph showing the removal mechanism
3.148 +
3.149 +{{{#!graphviz
3.150 +#format svg
3.151 +#transform notugly
3.152 +digraph provider_removal {
3.153 + node [fontsize="12.0",fontname="sans-serif",shape=box];
3.154 + edge [fontsize="12.0",fontname="sans-serif"];
3.155 + rankdir=LR;
3.156 +
3.157 + ResourceServer -> Resource [label="close"];
3.158 + Resource -> ProviderRegistry [label="detach"];
3.159 + ProviderRegistry -> Provider [label="remove"];
3.160 +}
3.161 +}}}
3.162 +
3.163 +########
3.164 +
3.165 +== Resources ==
3.166 +
3.167 +Resources are objects accessed by clients that support a basic level of
3.168 +accounting and management.
3.169 +
3.170 +The base interface of a resource is as follows:
3.171 +
3.172 +{{{
3.173 +void activate()
3.174 +void close()
3.175 +}}}
3.176 +
3.177 +Activation of a resource is an optional operation that performs any
3.178 +initialisation before a resource is made available to its user.
3.179 +
3.180 +In practice, other operations are required to make resources useful.
3.181 +
3.182 +In some cases, resources provide the mechanism by which each user of a
3.183 +filesystem object may access that object independently. They would then
3.184 +effectively provide a session in which accesses can occur.
3.185 +
3.186 +=== Directory Resources ===
3.187 +
3.188 +Directory resources primarily expose the contents of directories in the
3.189 +filesystem to a user. They employ directory accessors which concern themselves
3.190 +with the actual filesystem content.
3.191 +
3.192 +[[Components#Directories|Directory components]] are provided using directory
3.193 +resources.
3.194 +
3.195 +<<Anchor(Pager)>>
3.196 +=== Pagers or File Resources ===
3.197 +
3.198 +Pagers are resources that support dataspace access operations, thus allowing
3.199 +the resources to expose filesystem content in mapped memory regions to a
3.200 +particular user of the object providing the content.
3.201 +
3.202 +[[Components#Files|File components]] and [[Components#Pipes|pipe components]]
3.203 +are provided using pagers.
3.204 +
3.205 +=== Filesystem Resources ===
3.206 +
3.207 +Filesystem resources provide the entry point for access to a filesystem by
3.208 +other components or programs. Since filesystems often enforce identity-based
3.209 +access controls, a filesystem resource will typically support the
3.210 +`open_for_user` operation in various forms, with the result of this operation
3.211 +being the instantiation of an `OpenerResource` configured for the indicated
3.212 +user identity.
3.213 +
3.214 +== Registries ==
3.215 +
3.216 +The basic mechanism for obtaining a resource involves a registry, as
3.217 +illustrated by the following diagram.
3.218 +
3.219 +######## A graph showing the use of a registry in obtaining resources
3.220 +
3.221 +{{{#!graphviz
3.222 +#format svg
3.223 +#transform notugly
3.224 +digraph registry {
3.225 + node [fontsize="12.0",fontname="sans-serif",shape=box];
3.226 + edge [fontsize="12.0",fontname="sans-serif"];
3.227 + rankdir=LR;
3.228 +
3.229 + subgraph {
3.230 + node [label="OpenerContextResource"];
3.231 + rank=same;
3.232 +
3.233 + OpenerContextResource1 -> OpenerContextResource2 [dir=none,style=dotted];
3.234 + }
3.235 +
3.236 + subgraph {
3.237 + node [label="OpenerResource"];
3.238 + rank=same;
3.239 +
3.240 + OpenerResource1 -> OpenerResource2 [dir=none,style=dotted];
3.241 + }
3.242 +
3.243 + subgraph {
3.244 + rank=same;
3.245 +
3.246 + ResourceRegistry -> Resource;
3.247 + }
3.248 +
3.249 + subgraph {
3.250 + rank=same;
3.251 +
3.252 + ProviderRegistry -> Provider;
3.253 + }
3.254 +
3.255 + OpenerContextResource1 -> OpenerResource1 [label="open"];
3.256 + OpenerResource1 -> ResourceRegistry [label="get_resource"];
3.257 + ResourceRegistry -> ProviderRegistry [label="get/set"];
3.258 +
3.259 + Provider -> Resource -> OpenerResource2 -> OpenerContextResource2;
3.260 +}
3.261 +}}}
3.262 +
3.263 +########
3.264 +
3.265 +The `ResourceRegistry` coordinates access to filesystem resources and, through
3.266 +synchronisation, prevents conflicting operations from occurring concurrently.
3.267 +For example, a removal operation on a file may not be allowed to occur while
3.268 +an opening operation is in progress.
3.269 +
3.270 +To achieve this, a common lock is employed to serialise access, with any
3.271 +underlying filesystem operations being initiated from the registry while the
3.272 +lock is held. An object providing the `FileOpening` interface for a filesystem
3.273 +provides such operations, and the following interaction pattern is thereby
3.274 +employed:
3.275 +
3.276 +######## A graph showing the registry coordinating filesystem operations
3.277 +
3.278 +{{{#!graphviz
3.279 +#format svg
3.280 +#transform notugly
3.281 +digraph registry_operations {
3.282 + node [fontsize="12.0",fontname="sans-serif",shape=box];
3.283 + edge [fontsize="12.0",fontname="sans-serif"];
3.284 + rankdir=LR;
3.285 +
3.286 + OpenerContextResource -> OpenerResource -> ResourceRegistry -> FileOpening;
3.287 +}
3.288 +}}}
3.289 +
3.290 +########
3.291 +
3.292 +Since the `ResourceRegistry` functionality is generic, it could be specialised
3.293 +for each filesystem or be configured with an appropriate reference to a
3.294 +`FileOpening` object. The `OpenerResource` would then be generic, invoking the
3.295 +registry which would, in turn, invoke the opening object.
3.296 +
3.297 +However, the chosen approach is to permit `OpenerResource` objects to
3.298 +implement the `FileOpening` interface for each filesystem, meaning that the
3.299 +`ResourceRegistry` will end up being called by the opener and then invoking
3.300 +the opener in return. This is slightly more convenient than the alternative
3.301 +since the opener can be configured with a given user identity, and such
3.302 +identity details will ultimately be employed when accessing the underlying
3.303 +filesystem itself.
4.1 --- a/docs/wiki/Filesystems Sat Jun 11 18:29:28 2022 +0200
4.2 +++ b/docs/wiki/Filesystems Sun Jun 12 00:00:12 2022 +0200
4.3 @@ -5,6 +5,12 @@
4.4
4.5 == Topics ==
4.6
4.7 - * [[Components]] and [[Mechanisms]]
4.8 - * [[ClientLibrary|Client Library]]
4.9 + * [[ClientLibrary|Client Library]] - convenience functions and structures to
4.10 + access filesystem objects
4.11 + * [[Components]] - server objects exposed at the system level that support
4.12 + filesystem access
4.13 + * [[FilesystemAccess|Filesystem Access]] - objects within filesystem servers
4.14 + that manage filesystem resources and content
4.15 + * [[Paging]] - the mechanism by which filesystem content is provided for use
4.16 + by client programs
4.17 * [[Users]]
5.1 --- a/docs/wiki/Mechanisms Sat Jun 11 18:29:28 2022 +0200
5.2 +++ /dev/null Thu Jan 01 00:00:00 1970 +0000
5.3 @@ -1,299 +0,0 @@
5.4 -= Mechanisms =
5.5 -
5.6 -Within the filesystem server library, a number of different abstractions and
5.7 -mechanisms are employed to provide access to filesystem objects.
5.8 -
5.9 -<<TableOfContents(2,3)>>
5.10 -
5.11 -The abstractions or components used in the library are organised as follows.
5.12 -
5.13 -######## A graph showing the relationships between library components
5.14 -
5.15 -{{{#!graphviz
5.16 -#format svg
5.17 -#transform notugly
5.18 -digraph components {
5.19 - node [fontsize="12.0",fontname="sans-serif",shape=box];
5.20 - edge [fontsize="12.0",fontname="sans-serif"];
5.21 - rankdir=LR;
5.22 -
5.23 - subgraph {
5.24 - rank=same;
5.25 -
5.26 - Resource -> DirectoryResource -> FilePager
5.27 - [dir=none,style=dotted];
5.28 - }
5.29 -
5.30 - subgraph {
5.31 - rank=same;
5.32 -
5.33 - Provider -> DirectoryProvider -> FileProvider
5.34 - [dir=none,style=dotted];
5.35 - }
5.36 -
5.37 - subgraph {
5.38 - rank=same;
5.39 -
5.40 - Accessor -> Ext2FileAccessor
5.41 - [dir=none,style=dotted];
5.42 - }
5.43 -
5.44 - subgraph {
5.45 - rank=same;
5.46 -
5.47 - DirectoryAccessor -> Ext2DirectoryAccessor
5.48 - [dir=none,style=dotted];
5.49 - }
5.50 -
5.51 - subgraph {
5.52 - node [shape=ellipse];
5.53 - rank=same;
5.54 -
5.55 - Directory [label="dir"];
5.56 - File [label="dir/file"];
5.57 - }
5.58 -
5.59 - DirectoryResource -> DirectoryProvider -> Ext2DirectoryAccessor -> Directory;
5.60 - FilePager -> FileProvider -> Ext2FileAccessor -> File;
5.61 -}
5.62 -}}}
5.63 -
5.64 -########
5.65 -
5.66 -This document uses C-style interface syntax since the components operating
5.67 -within the described mechanisms will themselves be described using such
5.68 -syntax.
5.69 -
5.70 -== Accountable ==
5.71 -
5.72 -This interface provides the following operations:
5.73 -
5.74 -{{{
5.75 -void attach()
5.76 -unsigned int detach()
5.77 -}}}
5.78 -
5.79 -Its purpose is to provide reference counting for components, with the `attach`
5.80 -operation incrementing the number of users of a component, and with the
5.81 -`detach` operation decrementing the number of users and returning the
5.82 -resulting number of users. When components no longer have any attached users,
5.83 -they may perform operations to tidy up after themselves and permit their
5.84 -deallocation.
5.85 -
5.86 -== Accessors ==
5.87 -
5.88 -Accessors provide the means of accessing filesystem object data and metadata.
5.89 -Conceptually, files and directories are both exposed by accessors, although
5.90 -the interfaces and mechanisms may differ between these types of objects.
5.91 -
5.92 -=== Directory Accessors ===
5.93 -
5.94 -Currently, directory accessors provide support for traversing directory
5.95 -listings, obtaining the relevant filesystem metadata using the underlying
5.96 -filesystem access library.
5.97 -
5.98 -=== File Accessors ===
5.99 -
5.100 -File content is accessed through an interface with the following operations:
5.101 -
5.102 -{{{
5.103 -void close()
5.104 -void fill(Flexpage *flexpage)
5.105 -void flush(Flexpage *flexpage)
5.106 -offset_t get_size()
5.107 -void set_size(offset_t size)
5.108 -}}}
5.109 -
5.110 -The operations need to be supported with actual filesystem operations. For
5.111 -example, ext2-based filesystems employ a specific abstraction which invokes
5.112 -library functions provided by the `libext2fs` package.
5.113 -
5.114 -== Providers ==
5.115 -
5.116 -Providers encapsulate the essential functionality for accessing filesystem
5.117 -objects. Implementing the `Accountable` interface, they are shared by
5.118 -resources and discarded when no resources are using them.
5.119 -
5.120 -The following operations are supported by providers:
5.121 -
5.122 -{{{
5.123 -ProviderRegistry *registry()
5.124 -long make_resource(offset_t *size, object_flags_t *object_flags, Resource **resource)
5.125 -bool removal_pending()
5.126 -void remove()
5.127 -void remove_pending(bool remove)
5.128 -}}}
5.129 -
5.130 -Providers are associated with filesystem objects in a registry which can be
5.131 -obtained from each provider using the `registry` operation.
5.132 -
5.133 -Providers also support the creation of resources through which each user of a
5.134 -provider exercises its use of that provider. The `make_resource` operation
5.135 -performs the creation and returns size, flags and resource instance details.
5.136 -
5.137 -The removal of providers can be directed using the `remove_pending` and
5.138 -`remove` operations. Where `remove_pending` has been called with a true value
5.139 -as its parameter, the `removal_pending` operation will also return a true
5.140 -value, and upon the provider being discarded, the `remove` operation will be
5.141 -invoked. The `remove` operation performs the appropriate underlying operation
5.142 -to remove the object being provided.
5.143 -
5.144 -Typically, removal of providers is managed by the provider registry when
5.145 -resources are closed and detached from providers.
5.146 -
5.147 -######## A graph showing the removal mechanism
5.148 -
5.149 -{{{#!graphviz
5.150 -#format svg
5.151 -#transform notugly
5.152 -digraph provider_removal {
5.153 - node [fontsize="12.0",fontname="sans-serif",shape=box];
5.154 - edge [fontsize="12.0",fontname="sans-serif"];
5.155 - rankdir=LR;
5.156 -
5.157 - ResourceServer -> Resource [label="close"];
5.158 - Resource -> ProviderRegistry [label="detach"];
5.159 - ProviderRegistry -> Provider [label="remove"];
5.160 -}
5.161 -}}}
5.162 -
5.163 -########
5.164 -
5.165 -== Resources ==
5.166 -
5.167 -Resources are objects accessed by clients that support a basic level of
5.168 -accounting and management.
5.169 -
5.170 -The base interface of a resource is as follows:
5.171 -
5.172 -{{{
5.173 -void activate()
5.174 -void close()
5.175 -}}}
5.176 -
5.177 -Activation of a resource is an optional operation that performs any
5.178 -initialisation before a resource is made available to its user.
5.179 -
5.180 -In practice, other operations are required to make resources useful.
5.181 -
5.182 -In some cases, resources provide the mechanism by which each user of a
5.183 -filesystem object may access that object independently. They would then
5.184 -effectively provide a session in which accesses can occur.
5.185 -
5.186 -=== Directory Resources ===
5.187 -
5.188 -Directory resources primarily expose the contents of directories in the
5.189 -filesystem to a user. They employ directory accessors which concern themselves
5.190 -with the actual filesystem content.
5.191 -
5.192 -[[Components#Directories|Directory components]] are provided using directory
5.193 -resources.
5.194 -
5.195 -=== Pagers or File Resources ===
5.196 -
5.197 -Pagers are resources that support dataspace access operations, thus allowing
5.198 -the resources to expose filesystem content in mapped memory regions to a
5.199 -particular user of the object providing the content.
5.200 -
5.201 -[[Components#Files|File components]] and [[Components#Pipes|pipe components]]
5.202 -are provided using pagers.
5.203 -
5.204 -=== Filesystem Resources ===
5.205 -
5.206 -Filesystem resources provide the entry point for access to a filesystem by
5.207 -other components or programs. Since filesystems often enforce identity-based
5.208 -access controls, a filesystem resource will typically support the
5.209 -`open_for_user` operation in various forms, with the result of this operation
5.210 -being the instantiation of an `OpenerResource` configured for the indicated
5.211 -user identity.
5.212 -
5.213 -== Registries ==
5.214 -
5.215 -The basic mechanism for obtaining a resource involves a registry, as
5.216 -illustrated by the following diagram.
5.217 -
5.218 -######## A graph showing the use of a registry in obtaining resources
5.219 -
5.220 -{{{#!graphviz
5.221 -#format svg
5.222 -#transform notugly
5.223 -digraph registry {
5.224 - node [fontsize="12.0",fontname="sans-serif",shape=box];
5.225 - edge [fontsize="12.0",fontname="sans-serif"];
5.226 - rankdir=LR;
5.227 -
5.228 - subgraph {
5.229 - node [label="OpenerContextResource"];
5.230 - rank=same;
5.231 -
5.232 - OpenerContextResource1 -> OpenerContextResource2 [dir=none,style=dotted];
5.233 - }
5.234 -
5.235 - subgraph {
5.236 - node [label="OpenerResource"];
5.237 - rank=same;
5.238 -
5.239 - OpenerResource1 -> OpenerResource2 [dir=none,style=dotted];
5.240 - }
5.241 -
5.242 - subgraph {
5.243 - rank=same;
5.244 -
5.245 - ResourceRegistry -> Resource;
5.246 - }
5.247 -
5.248 - subgraph {
5.249 - rank=same;
5.250 -
5.251 - ProviderRegistry -> Provider;
5.252 - }
5.253 -
5.254 - OpenerContextResource1 -> OpenerResource1 [label="open"];
5.255 - OpenerResource1 -> ResourceRegistry [label="get_resource"];
5.256 - ResourceRegistry -> ProviderRegistry [label="get/set"];
5.257 -
5.258 - Provider -> Resource -> OpenerResource2 -> OpenerContextResource2;
5.259 -}
5.260 -}}}
5.261 -
5.262 -########
5.263 -
5.264 -The `ResourceRegistry` coordinates access to filesystem resources and, through
5.265 -synchronisation, prevents conflicting operations from occurring concurrently.
5.266 -For example, a removal operation on a file may not be allowed to occur while
5.267 -an opening operation is in progress.
5.268 -
5.269 -To achieve this, a common lock is employed to serialise access, with any
5.270 -underlying filesystem operations being initiated from the registry while the
5.271 -lock is held. An object providing the `FileOpening` interface for a filesystem
5.272 -provides such operations, and the following interaction pattern is thereby
5.273 -employed:
5.274 -
5.275 -######## A graph showing the registry coordinating filesystem operations
5.276 -
5.277 -{{{#!graphviz
5.278 -#format svg
5.279 -#transform notugly
5.280 -digraph registry_operations {
5.281 - node [fontsize="12.0",fontname="sans-serif",shape=box];
5.282 - edge [fontsize="12.0",fontname="sans-serif"];
5.283 - rankdir=LR;
5.284 -
5.285 - OpenerContextResource -> OpenerResource -> ResourceRegistry -> FileOpening;
5.286 -}
5.287 -}}}
5.288 -
5.289 -########
5.290 -
5.291 -Since the `ResourceRegistry` functionality is generic, it could be specialised
5.292 -for each filesystem or be configured with an appropriate reference to a
5.293 -`FileOpening` object. The `OpenerResource` would then be generic, invoking the
5.294 -registry which would, in turn, invoke the opening object.
5.295 -
5.296 -However, the chosen approach is to permit `OpenerResource` objects to
5.297 -implement the `FileOpening` interface for each filesystem, meaning that the
5.298 -`ResourceRegistry` will end up being called by the opener and then invoking
5.299 -the opener in return. This is slightly more convenient than the alternative
5.300 -since the opener can be configured with a given user identity, and such
5.301 -identity details will ultimately be employed when accessing the underlying
5.302 -filesystem itself.
6.1 --- /dev/null Thu Jan 01 00:00:00 1970 +0000
6.2 +++ b/docs/wiki/Paging Sun Jun 12 00:00:12 2022 +0200
6.3 @@ -0,0 +1,110 @@
6.4 += Paging =
6.5 +
6.6 +Within filesystem servers, a number of abstractions are combined to support
6.7 +[[FilesystemAccess|access to filesystem content]] through paging: the delivery
6.8 +of content to clients in mapped memory.
6.9 +
6.10 +The general paging mechanism used to support paging is depicted below.
6.11 +
6.12 +######## A graph showing the general paging mechanism
6.13 +
6.14 +{{{#!graphviz
6.15 +#format svg
6.16 +#transform notugly
6.17 +digraph paging {
6.18 + node [fontsize="12.0",fontname="sans-serif",shape=box];
6.19 + edge [fontsize="12.0",fontname="sans-serif"];
6.20 + rankdir=LR;
6.21 +
6.22 + subgraph {
6.23 + rank=same;
6.24 +
6.25 + Pager_note [shape=note,style=filled,fillcolor=gold,label="Provides\nfilesystem\ncontent"];
6.26 + Pager;
6.27 +
6.28 + Pager_note -> Pager [dir=none,style=dotted];
6.29 + }
6.30 +
6.31 + subgraph {
6.32 + rank=same;
6.33 +
6.34 + PageMapper_note [shape=note,style=filled,fillcolor=gold,label="Provides\npopulated\nfile pages"];
6.35 + PageMapper;
6.36 +
6.37 + PageMapper_note -> PageMapper [dir=none,style=dotted];
6.38 + }
6.39 +
6.40 + subgraph {
6.41 + rank=same;
6.42 +
6.43 + Accessor_note [shape=note,style=filled,fillcolor=gold,label="Populates\nfile pages"];
6.44 + Accessor;
6.45 +
6.46 + Accessor_note -> Accessor [dir=none,style=dotted];
6.47 + }
6.48 +
6.49 + AccessMap [shape=record,label="<head> AccessMap |
6.50 + { offset-0 |<fp0> flexpage-0 } | ... |
6.51 + { offset-n |<fpn> flexpage-n } | ..."];
6.52 +
6.53 + subgraph {
6.54 + rank=same;
6.55 +
6.56 + Pages_note [shape=note,style=filled,fillcolor=gold,label="Provides\nmemory\npages"];
6.57 + Pages;
6.58 +
6.59 + Pages_note -> Pages [dir=none,style=dotted];
6.60 + }
6.61 +
6.62 + PageQueue [shape=record,label="<head> PageQueue |
6.63 + { owner-0 |<fp0> flexpage-0 } | ... |
6.64 + { owner-N |<fpN> flexpage-N } | <end> ..."];
6.65 +
6.66 + Flexpage_offset_n [shape=record,label="<head> Flexpage |
6.67 + offset | size |<r> region"];
6.68 +
6.69 + subgraph {
6.70 + rank=same;
6.71 +
6.72 + Memory_note [shape=note,style=filled,fillcolor=gold,label="Allocates\nmemory\nregions"];
6.73 + Memory;
6.74 +
6.75 + Memory_note -> Memory [dir=none,style=dotted];
6.76 + }
6.77 +
6.78 + Region;
6.79 +
6.80 + /* Relationships. */
6.81 +
6.82 + PageMapper -> Accessor;
6.83 + PageMapper -> AccessMap:head;
6.84 + Pager -> PageMapper -> Pages;
6.85 + Pages -> Memory;
6.86 + Pages -> PageQueue:head;
6.87 + AccessMap:fpn -> Flexpage_offset_n:head;
6.88 + PageQueue:fpN -> Flexpage_offset_n:head;
6.89 + Flexpage_offset_n:r -> Region;
6.90 +}
6.91 +}}}
6.92 +
6.93 +########
6.94 +
6.95 +A [[FilesystemAccess#Pager|Pager]], being a resource that provides filesystem
6.96 +content to clients for a particular file, requests pages from the PageMapper,
6.97 +itself having the role of supplying populated pages of content from the given
6.98 +file. The PageMapper relies on the AccessMap to discover existing pages for
6.99 +any accessed region of a file, involving the Pages object (or page collection)
6.100 +to obtain new memory pages where existing pages are not available or are no
6.101 +longer valid, and requesting the population of such pages through use of the
6.102 +associated Accessor for the file concerned.
6.103 +
6.104 +The Pages object may either obtain new memory pages from a Memory object or
6.105 +reclaim (or recycle) existing pages from a PageQueue, this recording all pages
6.106 +that have been populated and supplied to clients. Since it may be desirable to
6.107 +limit the number of pages employed to satisfy file access operations, the
6.108 +PageQueue provides a kind of lending mechanism: pages are issued to clients
6.109 +and then added to the end of the queue; where no new page can be supplied by a
6.110 +Memory object, the issued page at the head of the queue is obtained for
6.111 +recycling; ultimately, an issued page will remain valid for its user (the
6.112 +client accessing its contents) until all pages in front of it in the queue are
6.113 +themselves recycled and it is then removed from the queue for recycling.