# HG changeset patch # User Paul Boddie # Date 1669913712 -3600 # Node ID ee511ba199afb0c6c3a5a7c66a168b1217fc2fe2 # Parent d431ace6aeaf9f41d0babef5dd4a85cb395cdbec Improved the documentation in a number of areas. diff -r d431ace6aeaf -r ee511ba199af docs/wiki/Client_Library --- a/docs/wiki/Client_Library Mon Nov 21 01:19:48 2022 +0100 +++ b/docs/wiki/Client_Library Thu Dec 01 17:55:12 2022 +0100 @@ -7,8 +7,8 @@ library, with the functions in the C library invoking client library functions and employing client library structures internally. -The client library is provided by `libfsclient` within the `departure` -package. +The client library is provided by [[Libraries#libfsclient|`libfsclient`]] +within the `departure` package. <> @@ -28,7 +28,7 @@ The members of the `file_t` data structure are as follows: -|| '''Field''' || '''Description''' || +|| '''Member''' || '''Description''' || || `ref` || A reference to the component providing file content || || `memory` || The memory address of the exposed file region || || `start_pos` || The start position of the region in the file || @@ -203,18 +203,74 @@ === Notifications === -Since files and pipes may be accessed by multiple clients, necessarily for any -sensible use of the latter, notifications can be configured to communicate a -change in state to other users of these resources when they are accessed. +Since files and pipes may be accessed by multiple clients, this being of +particular significance for any real use of pipes, notifications can be +configured to communicate a change in state to other users of these resources +when they are accessed. Directories can also be monitored using notifications. Notification types are specified using values encoding a number of flags, and the following flags are available for this purpose: || '''Flag''' || '''Notification Type''' || || `NOTIFY_CONTENT_AVAILABLE` || Content available to read || +|| `NOTIFY_FILE_OPENED` || File opened in directory || || `NOTIFY_PEER_CLOSED` || Other party has closed their endpoint || || `NOTIFY_SPACE_AVAILABLE` || Space available for writing || +The delivery of notifications is requested by subscribing to notifications for +a given resource via a notifier object: + +{{{ +long client_subscribe(file_t *file, notify_flags_t flags, file_notifier_t *notifier); +}}} + +A notifier object can be common throughout all threads in a task, being +obtained using the following function: + +{{{ +file_notifier_t *client_notifier_task(); +}}} + +Alternatively, a local notifier can be created for use within a thread: + +{{{ +file_notifier_t *client_notifier_local(); +}}} + +Local notifiers must be closed when they are no longer needed: + +{{{ +void client_notifier_close(file_notifier_t *notifier); +}}} + +When notifications are no longer needed, an unsubscribe operation can be +invoked: + +{{{ +long client_unsubscribe(file_t *file, file_notifier_t *notifier); +}}} + +==== Example ==== + +{{{ +file_notifier_t *notifier = client_notifier_local(); +file_t *directory = client_open(filename, O_DIRECTORY); + +if (client_opened(directory)) +{ + if (!client_subscribe(directory, NOTIFY_FILE_OPENED, notifier)) + { + if (!client_wait_file(directory, notifier)) + { + /* File opened in directory. */ + } + } +} + +client_close(directory); +client_notifier_close(notifier); +}}} + === Blocking Operations === Reading and writing operations can be configured to block if data cannot be @@ -228,3 +284,12 @@ For pipes, blocking behaviour is the default and must be disabled explicitly, either by opening using the `O_NONBLOCK` flag or by calling `client_set_blocking` with no flags set. + +Blocking behaviour is supported using the notification functionality. When +access to a file or pipe cannot be satisfied for a particular operation, such +as reading not being able to yield more content or writing not being able to +submit more content, the task-level notifier will be used to wait for +notifications applicable to the file or pipe involved. Consequently, the +access will effectively block until notifications are delivered indicating +that the state of the file or pipe has changed, and until it is determined +that the change of state will allow the operation to proceed successfully. diff -r d431ace6aeaf -r ee511ba199af docs/wiki/Components --- a/docs/wiki/Components Mon Nov 21 01:19:48 2022 +0100 +++ b/docs/wiki/Components Thu Dec 01 17:55:12 2022 +0100 @@ -6,8 +6,9 @@ mechanisms for opening, reading, writing, and closing files, together with various other operations. -Components are provided by functionality in `libfsserver` used by programs -found in the `servers` directory within the `departure` package. +Components are provided by functionality in [[Libraries#libfsserver| +`libfsserver`]] used by programs found in the `servers` directory within the +`departure` package. <> @@ -16,145 +17,6 @@ document are described using excerpts from the appropriate interface descriptions. -== Overview == - -An overview of the component interactions involved in opening a file or -directory is provided by the diagram below. - -######## A graph showing the interactions between components - -{{{#!graphviz -#format svg -#transform notugly -digraph components { - node [fontsize="12.0",fontname="sans-serif",shape=box]; - edge [fontsize="12.0",fontname="sans-serif"]; - rankdir=LR; - - subgraph { - node [label="Client"]; - rank=min; - - Client1; Client2; Client3; Client4; Client5; Client6; Client7; - } - - subgraph { - rank=same; - - Memory [label="filename",shape=note]; - } - - subgraph { - rank=max; - - Filesystem; - - subgraph { - node [label="Opener\n(user)"]; - Opener1; Opener2; - } - - subgraph { - node [label="OpenerContext"]; - OpenerContext1; OpenerContext2; OpenerContext3; - } - - Object [label="MappedFile\nor\nDirectory"]; - } - - Client1 -> Client2 -> Client3 -> Client4 -> Client5 -> Client6 -> Client7 [dir=none,style=dotted]; - Opener1 -> Opener2 [dir=none,style=dotted]; - OpenerContext1 -> OpenerContext2 -> OpenerContext3 [dir=none,style=dotted]; - - Client1 -> Filesystem [label="open_for_user(user)"]; - Filesystem -> Opener1; - Opener1 -> Client2; - - Client3 -> Opener2 [label="context()"]; - Opener2 -> OpenerContext1; - OpenerContext1 -> Client4; - - Client5 -> Memory -> OpenerContext2; - - Client6 -> OpenerContext3 [label="open(flags, ...)"]; - OpenerContext3 -> Object; - Object -> Client7; -} -}}} - -######## - -In pseudocode, the operations as conducted by the client program are as -follows: - -{{{ -opener = filesystem.open_for_user(user) -context = opener.context() -context.write("filename") # this being a memory access operation -file = context.open(flags, ...) -}}} - -Reading from an opened directory is achieved as shown in the following -diagram. - -######## A graph showing the interactions between components - -{{{#!graphviz -#format svg -#transform notugly -digraph components { - node [fontsize="12.0",fontname="sans-serif",shape=box]; - edge [fontsize="12.0",fontname="sans-serif"]; - rankdir=LR; - - subgraph { - node [label="Client"]; - rank=min; - - Client1; Client2; Client3; Client4; - } - - subgraph { - rank=same; - - Memory [label="entries",shape=note]; - } - - subgraph { - rank=max; - - Directory; - - subgraph { - node [label="Reader"]; - - Reader1; Reader2; Reader3; - } - } - - Client1 -> Client2 -> Client3 -> Client4 [dir=none,style=dotted]; - Reader1 -> Reader2 -> Reader3 [dir=none,style=dotted]; - - Client1 -> Directory [label="opendir()"]; - Directory -> Reader1; - Reader1 -> Client2; - - Client3 -> Reader2 [label="current_region()"]; - Reader3 -> Memory -> Client4; -} -}}} - -######## - -In pseudocode, the operations as conducted by the client program are as -follows: - -{{{ -reader = directory.opendir() -reader.current_region() -entries = reader.read() # this being a memory access operation -}}} - == Filesystems == Filesystems implement the `Filesystem` interface which provides the @@ -167,6 +29,52 @@ The operation yields a file opener appropriate for the given [[Users|user]] credentials. +######## A graph showing the interactions between components + +{{{#!graphviz +#format svg +#transform notugly +digraph open_for_user { + node [fontsize="12.0",fontname="sans-serif",shape=box]; + edge [fontsize="12.0",fontname="sans-serif"]; + rankdir=LR; + + subgraph { + node [label="Client"]; + rank=min; + + Client1; Client2; + } + + subgraph { + rank=max; + + Filesystem; + + subgraph { + node [label="Opener\n(user)"]; + Opener; + } + } + + Client1 -> Client2 [dir=none,style=dotted]; + + Client1 -> Filesystem [label="open_for_user(user)"]; + Filesystem -> Opener; + Opener -> Client2; + +} +}}} + +######## + +In pseudocode, the operations as conducted by the client program are as +follows: + +{{{ +opener = filesystem.open_for_user(user) +}}} + ((Openers)) == File and Directory Openers == @@ -180,6 +88,54 @@ Each client program, task or thread obtains its own context because it will need its own dedicated channel for communication with the filesystem. +######## A graph showing the interactions between components + +{{{#!graphviz +#format svg +#transform notugly +digraph context { + node [fontsize="12.0",fontname="sans-serif",shape=box]; + edge [fontsize="12.0",fontname="sans-serif"]; + rankdir=LR; + + subgraph { + node [label="Client"]; + rank=min; + + Client1; Client2; + } + + subgraph { + rank=max; + + subgraph { + node [label="Opener\n(user)"]; + Opener; + } + + subgraph { + node [label="OpenerContext"]; + OpenerContext; + } + } + + Client1 -> Client2 [dir=none,style=dotted]; + + Client1 -> Opener [label="context()"]; + Opener -> OpenerContext; + OpenerContext -> Client2; +} +}}} + +######## + +In pseudocode, the operations as conducted by the client program are as +follows: + +{{{ +context = opener.context() +}}} + == Opener Contexts == An opener context acts as a dataspace, meaning that it can be attached to a @@ -205,6 +161,61 @@ Alongside regular files, directories may also be opened. Reading from them yields a listing of directory entries. +######## A graph showing the interactions between components + +{{{#!graphviz +#format svg +#transform notugly +digraph open { + node [fontsize="12.0",fontname="sans-serif",shape=box]; + edge [fontsize="12.0",fontname="sans-serif"]; + rankdir=LR; + + subgraph { + node [label="Client"]; + rank=min; + + Client1; Client2; Client3; + } + + subgraph { + rank=same; + + Memory [label="filename",shape=note]; + } + + subgraph { + rank=max; + + subgraph { + node [label="OpenerContext"]; + OpenerContext1; OpenerContext2; + } + + Object [label="MappedFile\nor\nDirectory"]; + } + + Client1 -> Client2 -> Client3 [dir=none,style=dotted]; + OpenerContext1 -> OpenerContext2 [dir=none,style=dotted]; + + Client1 -> Memory -> OpenerContext1; + + Client2 -> OpenerContext2 [label="open(flags, ...)"]; + OpenerContext2 -> Object; + Object -> Client3; +} +}}} + +######## + +In pseudocode, the operations as conducted by the client program are as +follows: + +{{{ +context.write("filename") # this being a memory access operation +file = context.open(flags, ...) +}}} + === Removing === Filesystem objects are removed by invoking the `remove` operation on an opener @@ -292,6 +303,67 @@ supporting precisely the same navigation mechanisms as those supported by files. +Reading from an opened directory is achieved as shown in the following +diagram. + +######## A graph showing the interactions between components + +{{{#!graphviz +#format svg +#transform notugly +digraph components { + node [fontsize="12.0",fontname="sans-serif",shape=box]; + edge [fontsize="12.0",fontname="sans-serif"]; + rankdir=LR; + + subgraph { + node [label="Client"]; + rank=min; + + Client1; Client2; Client3; Client4; + } + + subgraph { + rank=same; + + Memory [label="entries",shape=note]; + } + + subgraph { + rank=max; + + Directory; + + subgraph { + node [label="Reader"]; + + Reader1; Reader2; Reader3; + } + } + + Client1 -> Client2 -> Client3 -> Client4 [dir=none,style=dotted]; + Reader1 -> Reader2 -> Reader3 [dir=none,style=dotted]; + + Client1 -> Directory [label="opendir()"]; + Directory -> Reader1; + Reader1 -> Client2; + + Client3 -> Reader2 [label="current_region()"]; + Reader3 -> Memory -> Client4; +} +}}} + +######## + +In pseudocode, the operations as conducted by the client program are as +follows: + +{{{ +reader = directory.opendir() +reader.current_region() +entries = reader.read() # this being a memory access operation +}}} + == Pipe Openers == Distinct from filesystems but potentially used by them, pipe openers provide a diff -r d431ace6aeaf -r ee511ba199af docs/wiki/Filesystem_Access --- a/docs/wiki/Filesystem_Access Mon Nov 21 01:19:48 2022 +0100 +++ b/docs/wiki/Filesystem_Access Thu Dec 01 17:55:12 2022 +0100 @@ -18,10 +18,17 @@ subgraph { rank=same; - Opener_note [shape=note,style=filled,fillcolor=gold,label="Exposes\nfile open\noperation"]; + Filesystem_note [shape=note,style=filled,fillcolor=gold,label="Configures opener\ncomponents"]; + Filesystem; + + Filesystem_note -> Filesystem [dir=none,style=dotted]; + + open_for_user [fontsize="10.0",shape=ellipse]; + Opener; + Opener_note [shape=note,style=filled,fillcolor=gold,label="Exposes\nfile open\noperation"]; - Opener_note -> Opener [dir=none,style=dotted]; + Opener -> Opener_note [dir=none,style=dotted]; } subgraph { @@ -79,6 +86,11 @@ Accessor_note -> Accessor [dir=none,style=dotted]; } + /* Configuring an opener. */ + + Filesystem -> open_for_user [dir=none]; + open_for_user -> Opener; + /* Opening a file. */ Opener -> ResourceRegistry -> Resource; @@ -99,6 +111,9 @@ ######## +Firstly, an `Opener` must be obtained from a `Filesystem`, this configuring +the `Opener` for a particular user identity. + An `Opener` requests a `Resource` from a `ResourceRegistry`, each `Resource` acting as a [[ServerLibrary#Pager|`Pager`]] and providing the actual access mechanism to file content for a particular program. Since many programs may @@ -518,6 +533,7 @@ ResourceServer -> Resource [label="close"]; Resource -> Provider [label="notify_others\nunsubscribe"]; Resource -> ProviderRegistry [label="detach"]; + ProviderRegistry -> Provider [label="detach\ndelete"]; } }}} diff -r d431ace6aeaf -r ee511ba199af docs/wiki/Server_Library --- a/docs/wiki/Server_Library Mon Nov 21 01:19:48 2022 +0100 +++ b/docs/wiki/Server_Library Thu Dec 01 17:55:12 2022 +0100 @@ -3,8 +3,8 @@ Within the filesystem server library, a number of different abstractions and mechanisms are employed to provide access to filesystem objects. -The server library is provided by `libfsserver` within the `departure` -package. +The server library is provided by [[Libraries#libfsserver|`libfsserver`]] +within the `departure` package. <> @@ -12,9 +12,10 @@ within the described mechanisms will themselves be described using such syntax. -== Accountable == +((Accountable)) +== Accountables == -This interface provides the following operations: +The `Accountable` interface provides the following operations: {{{ void attach(); @@ -28,6 +29,9 @@ they may perform operations to tidy up after themselves and permit their deallocation. +The [[#Provider|`Provider`]] abstraction employs this interface to record its +usage by multiple resources. + == Accessors == Accessors provide the means of accessing filesystem object data and metadata. @@ -40,9 +44,22 @@ listings, obtaining the relevant filesystem metadata using the underlying filesystem access library. +The `DirectoryAccessor` interface provides the following operation: + +{{{ +void read_directory(file_t *writer); +}}} + +The `read_directory` operation is presented with a writer object into which +directory listing data will be written. In the case of the ext2 directory +accessor, the writer is presented to a directory iterator which traverses the +directory data structure, invoking a callback sending directory entries via +the writer to the client. + === File Accessors === -File content is accessed through an interface with the following operations: +File content is accessed through the `Accessor` interface with the following +operations: {{{ void close(); @@ -54,15 +71,17 @@ The operations need to be supported with actual filesystem operations. For example, ext2-based filesystems employ a specific abstraction which invokes -library functions provided by the `libext2fs` package. +library functions provided by [[Libraries#libext2fs|`libext2fs`]]. +((Provider)) == Providers == Providers encapsulate the essential functionality for accessing filesystem objects. Implementing the `Accountable` interface, they are shared by resources and discarded when no resources are using them. -The following operations are supported by providers: +The following operations are supported by providers as defined by the +`Provider` interface: {{{ ProviderRegistry *registry(); @@ -71,23 +90,34 @@ void remove_pending(bool remove); }}} -Providers are associated with filesystem objects in a registry which can be -obtained from each provider using the `registry` operation. +=== Origin and Ownership === + +A provider is created to represent a filesystem object when an attempt is made +to open that object. It is then recorded in a provider registry so that +subsequent attempts to open the object yield a common provider instance. The +provider registry having ownership of each provider can be obtained using its +`registry` operation. + +See the [[Filesystem Access#Opening Files|file opening mechanism]] for details +of the creation and registration of providers. + +=== Resource Creation === Providers also support the creation of resources through which each user of a provider exercises its use of that provider. The `make_resource` operation performs the creation and returns size, flags and resource instance details. -The removal of providers can be directed using the `remove_pending` operation. -Where `remove_pending` has been called with a true value as its parameter, the -`removal_pending` operation will also return a true value, and upon the -provider being discarded, a removal operation will be invoked on the -underlying object being provided. +See the [[Filesystem Access#Opening Files|file opening mechanism]] for details +of resource creation. + +=== Deallocation === -Typically, removal of providers is managed by the provider registry when -resources are closed and detached from providers. +Deallocation of providers is managed by the provider registry when resources +are closed and detached from providers. Since the registry effectively has +ownership of the provider, having registered it, the registry must therefore +be involved in its deallocation, deregistering it first. -######## A graph showing the removal mechanism +######## A graph showing the deallocation mechanism {{{#!graphviz #format svg @@ -99,28 +129,56 @@ ResourceServer -> Resource [label="close"]; Resource -> ProviderRegistry [label="detach"]; - ProviderRegistry -> Provider [label="delete"]; + ProviderRegistry -> Provider [label="detach\ndelete"]; } }}} ######## +=== Filesystem Object Removal === + +The removal of filesystem objects that are being represented by providers can +be directed using the `remove_pending` operation. Where `remove_pending` has +been called with a true value as its parameter, the `removal_pending` +operation will also return a true value, and upon the provider being +discarded, a removal operation will be invoked on the underlying object being +provided. + +See the [[Filesystem Access#Removing Files|file removal mechanism]] for more +details of the invocations involved. + +((Resource)) == Resources == Resources are objects accessed by clients that support a basic level of -accounting and management. +accounting and management. They act as servers and receive messages via the +interprocess communication (IPC) mechanism. -The base interface of a resource is as follows: +The `Resource` abstraction is intended to work with lower-level mechanisms +provided by [[Libraries#libipc|`libipc`]] involving data structures and +functions that are usable in the C programming language. This abstraction +integrates the fundamental `libipc` support with C++. + +The generic operations of the `Resource` interface are as follows: {{{ void activate(); void close(); }}} -Activation of a resource is an optional operation that performs any -initialisation before a resource is made available to its user. +The activation of a resource, supported by `activate`, is an optional +operation that performs any initialisation before a resource is made available +as a server. + +The `close` operation is invoked when resources are to be discarded. -In practice, other operations are required to make resources useful. +See the [[Filesystem Access#Closing Files|file closing mechanism]] for the +context in which the `close` operation is invoked. + +In practice, other operations are required to make resources useful. Such +other operations are provided by classes inheriting from `Resource` and thus +specialising it. Such classes also inherit from IPC interface types so as to +be able to support the invocation of operations exposed by such interfaces. In some cases, resources provide the mechanism by which each user of a filesystem object may access that object independently. They would then @@ -154,6 +212,46 @@ being the instantiation of an `OpenerResource` configured for the indicated user identity. +=== Server Framework Integration === + +Resources must also support specific operations for integration with the +lower-level IPC handling implemented in `libipc`. The following operations +will be invoked by the server framework in `libfsserver` to configure a server +controlled by `libipc`: + +{{{ +int expected_items(); +ipc_server_handler_type handler(); +void *interface(); +}}} + +The `expected_items` operation returns the number of message items expected +from clients invoking server operations exposed via IPC. Typically, the +returned value will be calculated for a given server interface by a tool such +as `idl`, provided by the idl4re distribution. + +The `handler` operation returns a reference to a handler function able to +interpret the operation codes in incoming messages and to dispatch to the +appropriate operations provided by the resource. Typically, such a function +will be generated by a tool such as `idl`. + +The `interface` operation returns a pointer to the resource instance, coerced +to an appropriate type for the handler function. Such a type will be that of +the most specialised IPC interface type inherited by the resource +implementation. The function returned by the `handler` operation will expect a +pointer to an instance of this appropriate type. + +For example, consider the following: + +{{{ +class FilePager : public Pager, public MappedFileObject +}}} + +Here, the `interface` operation will return a pointer coerced to +`MappedFileObject`, and the `handler` operation will return a reference to a +function expecting to be able to interpret this pointer as referring to +precisely that type. + == Registries == The basic mechanism for obtaining a resource involves a registry, as diff -r d431ace6aeaf -r ee511ba199af docs/wiki/Users --- /dev/null Thu Jan 01 00:00:00 1970 +0000 +++ b/docs/wiki/Users Thu Dec 01 17:55:12 2022 +0100 @@ -0,0 +1,72 @@ += Users = + +Since filesystems such as ext2 employ the concepts of users and groups, and +since access to such filesystems might be expected to respect the recorded +user and group metadata, permitting or denying access to objects as +appropriate, the need arises to define a user identity to control access to a +filesystem server and the filesystem objects it exposes. + +== Opener Configuration == + +Consequently, a filesystem server may not provide direct access to a +filesystem. Instead, it may only expose the [[Components#Filesystems| +`Filesystem`]] interface which provides the `open_for_user` operation. This +operation is used to configure an [[Components#Openers|`Opener`]] that +provides the actual interface for filesystem access as performed by a +particular user. + +Since the `open_for_user` operation involves the indication of an arbitrary +user identity, a server providing the `Filesystem` interface should only be +exposed to appropriately privileged components. An `Opener` obtained from the +operation can then be presented to a less privileged component. + +== User Structure == + +Ordinarily, user information is exchanged using a `user_t` structure defined +in [[Libraries#libsystypes|`libsystypes`]] with the following members: + +|| '''Member''' || '''Description''' || +|| `uid` || User identifier || +|| `gid` || Group identifier || +|| `umask` || File mode creation mask || + +The information broadly follows that of a traditional Unix system. Other +information, such as supplementary groups might conceivably be provided to the +filesystem server separately. Indeed, the user structure might be simplified, +removing the primary group information and providing this separately, too. + +== Opener Configuration in Ned == + +The following example illustrates the configuration of an opener and the +provision of the opener to a new task in the Lua-based scripting environment +of the Ned component in L4Re: + +{{{ +-- Obtain user filesystems with umask 0022 (18). + +local open_for_user = 6; +local ext2svr_paulb = L4.cast(L4.Proto.Factory, ext2svr):create(open_for_user, 1000, 1000, 18); + +l:startv({ + caps = { + server = ext2svr_paulb, + }, + log = { "client", "g" }, + }, + -- program, file to create + "rom/dstest_file_client", "home/paulb/new file"); +}}} + +Here, `ext2svr_paulb` is an opener configured for the user `paulb` who has +user and group identifiers of 1000. Since the Lua environment emphasises the +L4Re factory mechanism, and since factory operations involve the use of L4Re +variable-sized arguments ("vargs") as parameters, the signature of the factory +version of the operation consists of the individual elements of the user +abstraction: + +{{{ +open_for_user(in ipc_varg_sys_uid_t uid, + in ipc_varg_sys_gid_t gid, + in ipc_varg_sys_mode_t umask, + out cap opener) +}}}